{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 22:  Training Non-Linear Classifiers with Decision Tree Techniques\n",
    "\n",
    "## 22.1 Automated Learning of Logical Rules\n",
    "\n",
    "A lightbulb hangs above a stairwell. The bulb is connected to two switches. When both switches turned are off, the bulb stays off. If either switch is turned on, then the bulb will shine. However, if both switches are flipped on, then the bulb will turn off. Using the material from the previous two sections, can we train a classifier to learn this relationship? We'll find out by visualizing the problem in 2D space.\n",
    "\n",
    "**Listing 22. 1. Plotting the two-switch problem in 2D space**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAY80lEQVR4nO3df5Ac5X3n8fdHWgtlkR0Ra4XRD4xykS2rLpBDC4hUbMl2cgguV7IuvjowhDLOReFiYa1lV6DyAxJwqpLKsRbEdhRBKRSFbV1yRkYkwiSxI8mHkU8LZQvLBNdarqBFxFrZFjJSkLLMN390LxlGM7uzq+kezT6fV9XUTnc/2/19ENWf7V9PKyIwM7N0TWt3AWZm1l4OAjOzxDkIzMwS5yAwM0ucg8DMLHFd7S5goubMmRMXXXTRa9MHDx7k8OHDzJ07l4ULF57x+lu9PjOzeo4cgYMHoVIpbhvTpsHChTBnDjz11FNHIqKnbsOI6KjPsmXLolqlUom+vr4Aoq+vLyqVSkxGq9ZjZjaegwcjzj03Aor/zJoVMTQUAQxEg/1qxx0R1JJEf38/ABs3bgSgv78fSU2vIyLYsGEDGzdupK+vb8K/b2Y2ETfcACdPlrOtV16B668fu03HBwGcWRg4BMysTHv3Zp+RkXK2NzKSbQ9mdTdqMyWCACYXBg4BMyvb3Xdnf6WXKdveW85vtHzKBAFMLAwcAmZWtqNH4ZFHir1AXE+2vTed12j5lAoCaC4MHAJm1g5PPgkzZpR/RJCJhvFTWBBI2gL8MnA4Iv5jneUC7gGuAU4AH4yIp1u07YZh4BAws3bZuxdOnGjX1tXwubEijwgeAD4FPNhg+dXA4vxzBfBn+c+WaBQGDgEza5ddu8q7SHy6xju7woIgInZLumiMJquBByMigD2SZku6ICJebFUNtWEwGggOATNrh0OH2l1Bfe0cYmI+cLBqeiifdxpJayUNSBoYHh6e0Eaqw2CUQ8DM2uHUqXZXUF87g6DenrjuW3IiYnNE9EZEb09P/SekGxm9JlBtw4YNhF/IY2YlmzGj3RXU184gGAKqB/NZALT0wKn2wnClUqGvr4+NGzc6DMysdPPmtbuC+tp5++h2YJ2krWQXiV9q5fWBRncHnelwFGZmk7ViBeze3a4Lxo3/8i3y9tHPAyuBOZKGgDuAN2TlxCZgB9mto4Nkt4/e1Kptj3WLqMPAzNrlssuguxuOHWvH1tvwHEFEXDfO8gA+XMB2x71F1GFgZu1w5ZXtvGDcnucISjeRh8UcBmZWttmzYfVq+Ku/KneYiWnToFI59qNGy6dMEEzmiWGHgZmV7eMfh0cfLfcJ45kz4cSJf/5+o+VTIgjOZNgIh4GZlam3N7tW8MQT5Vw07urKtrdr18sNo6fjg6AVYwc5DMysTA89BEuWlBMEM2fCZz8LCxY0btPRQdDKAeQcBmZWlgUL4N574ZZbij1F1N0N99wD8+uO2fDvOjYIihhF1GFgZmW56SbYtw/uvx+OH2/9+s89F3791+FDHxq/bUcGQZFDSTsMzKwMEnzyk9n3++5r7ZFBd3cWAjXDrDXUkUFQ9FDSDgMzK8NoGFx8Maxfn72w5kyuG3R1ZdcE7rmnuSOB10RER33mzp0bQPT19UWlUokiVSqV6OvrK217ZpaugwcjVqyI6O6OmDYtIhsSornPtGnZ761YETE0VH/9wEA02K923BHB4cOHS3ufQO2RwaxZs7jrrrsK3aaZpWnBAti5EwYGslM627bBOedk1w/qHSV0dWXXAU6ehDVrYMOG7NbUyVB02Aic8+bNixdeeKHU0zQRwe23386aNWu49NJLS9uumaXr6FHYsyd7veXOnfDii9lO/5xz4IILYOXK7PmA5cuzJ5bHI+mpiKgbFR0XBL29vTEwMNDuMszMOspYQdDO9xGYmdlZwEFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIKDQJJqyQ9J2lQ0m11lv+kpEclfVPSfkk3FVmPmZmdrrAgkDQd+DRwNbAUuE7S0ppmHwa+HRGXACuBuyXNKKomMzM7XZFHBJcDgxFxICJOAVuB1TVtAnijJAGzgB8CIwXWZGZmNYoMgvnAwarpoXxetU8B7wAOAc8A6yOiUrsiSWslDUgaGB4eLqpeM7MkFRkEqjMvaqavAr4BzAN+DviUpDed9ksRmyOiNyJ6e3p6Wl2nmVnSigyCIWBh1fQCsr/8q90EPByZQeB7wJICazIzsxpFBsFeYLGkRfkF4GuB7TVtngfeCyDpfODtwIECazIzsxpdRa04IkYkrQMeB6YDWyJiv6Sb8+WbgLuAByQ9Q3Yq6daIOFJUTWZmdrrCggAgInYAO2rmbar6fgj4z0XWYGZmY/OTxWZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklrtAgkLRK0nOSBiXd1qDNSknfkLRf0q4i6zEzs9N1FbViSdOBTwO/BAwBeyVtj4hvV7WZDXwGWBURz0uaW1Q9ZmZWX5FHBJcDgxFxICJOAVuB1TVtPgA8HBHPA0TE4QLrMTOzOooMgvnAwarpoXxetbcB50naKekpSTfWW5GktZIGJA0MDw8XVK6ZWZomFQSSbm+mWZ15UTPdBSwD/gtwFfB7kt522i9FbI6I3ojo7enpmXC9ZmbW2GSPCP5nE22GgIVV0wuAQ3XafCkijkfEEWA3cMkkazIzs0loGASSjjX4/BiY18S69wKLJS2SNAO4Fthe0+YR4J2SuiR1A1cAz06yL2ZmNglj3TV0FLgsIr5fu0DSwdObv15EjEhaBzwOTAe2RMR+STfnyzdFxLOSvgTsAyrA/RHxrUn0w8zMJmmsIHgQeCtwWhAAn2tm5RGxA9hRM29TzfSfAH/SzPrMzKz1GgZBRPzuGMtuLaYcMzMrm4eYMDNLnIPAzCxxDgIzs8Q1NdZQPm7Q+dXtR4eFMDOzzjZuEEi6BbiD7O6hSj47gIsLrMvMzErSzBHBeuDtEfGDoosxM7PyNXON4CDwUtGFmJlZezQ8IpC0If96ANgp6W+Ak6PLI6K/4NrMzKwEY50aemP+8/n8MyP/mJnZFDLWk8V/UGYhZmbWHuNeI5D0d/krJUenz5P0eKFVmZlZaZq5WNwTEUdHJyLiR4DfLWxmNkU0EwSvSrpwdELSWzn9TWNmZtahmnmO4HeA/ydpVz79LuA3iivJzMzKNG4QRMSXJF0KLCd7D/FH89dKmpnZFNDMxeIvR8SRiPjriHg0Io5I+nIZxZmZWfHGeqBsJtANzJF0HtnRAMCbaO6dxWZm1gHGOjX0G0Af2U7/6ar5x4BPF1iTmZmVaKwHyu4B7pF0S0T8aYk1mZlZicY6NfSeiPgK8IKk/1a7PCIeLrQyMzMrxVinhlYAXwH+a51lATgIzMymgLFODd2R/7ypvHLMzKxszbyh7LvAHuCrwO6I+HbhVZmZWWmaGWJiKfDnwJuB/y3pgKRtxZZlZmZlaWqsIeBf858VsncXHy6yKDMzK08zYw0dA54B+oH7/O5iM7OppZkjguuA3cBvAlsl/YGk9xZblpmZlaWZQeceAR6RtAS4muxp498CfqLY0szMrAzNDDr3hfzOoXuAWcCNwHlFF2ZmZuVo5hrBHwFPR8SrRRdjZmbla3hEIOkySW+JiL0R8aqkGyU9IuleST/VzMolrZL0nKRBSbeNs61XJb1/Mp0wM7PJG+vU0J8DpwAkvYvsyOBB4CVg83grljSdbJTSq8meRbhO0tIG7f4YeHyixZuZ2ZkbKwimR8QP8+//A9gcEV+IiN8DfqaJdV8ODEbEgYg4BWwFVtdpdwvwBfxsgplZW4wZBJJGryG8l2wAulHNXFuYDxysmh7K571G0nxgDbBprBVJWitpQNLA8PBwE5s2M7NmjRUEnwd2SXoE+BeysYaQ9DNkp4fGozrzomZ6I3DreBeiI2JzRPRGRG9PT08TmzYzs2aNNfroH+bvJr4A+NuIGN2JTyM7nTOeIWBh1fQC4FBNm16yh9QA5gDXSBqJiC82V76ZmZ2pMU/xRMSeOvO+0+S69wKLJS0CXgCuBT5Qs65Fo98lPQD8tUPAzKxczZzrn5SIGJG0juxuoOnAlojYL+nmfPmY1wXMzKwchQUBQETsAHbUzKsbABHxwSJrMTOz+poZdM7MzKYwB4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIcBGZmiXMQmJklzkFgZpY4B4GZWeIKDQJJqyQ9J2lQ0m11ll8vaV/++ZqkS4qsx8zMTldYEEiaDnwauBpYClwnaWlNs+8BKyLiYuAuYHNR9ZiZWX1FHhFcDgxGxIGIOAVsBVZXN4iIr0XEj/LJPcCCAusxM7M6igyC+cDBqumhfF4jvwY8Vm+BpLWSBiQNDA8Pt7BEMzMrMghUZ17UbSi9mywIbq23PCI2R0RvRPT29PS0sEQzM+sqcN1DwMKq6QXAodpGki4G7geujogfFFiPmZnVUeQRwV5gsaRFkmYA1wLbqxtIuhB4GPjViPhOgbWYmVkDhR0RRMSIpHXA48B0YEtE7Jd0c758E3A78GbgM5IARiKit6iazMzsdIqoe9r+rNXb2xsDAwPtLsPMrKNIeqrRH9p+stjMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxDkIzMwS5yAwM0ucg8DMLHEOAjOzxBX2zuJ2O3oUnnwS9u6FXbvg0CE4dQpmzIB582DFCrjsMrjySpg9u93VmlkKnn76abZt28add95J/p72UuSvJJ7XaPmUC4KBAbj7bvjiF7Od/okTMDLy+jb/+I+wezd0d2fh8L73wcc+Br113+ZpZtYa27Zt4xOf+AQvv/wy/f39pYRBRLBhwwaACxq1mTJBMDQEN9yQHQG88gpUKtnPRkZG4Nix7Ptf/iVs354dITz0ECxYUE7NZpaWO++8k5dffpmNGzcCFB4GoyGQb+9wo3Ydf40gArZsgSVL4IknsiOASmVi66hUst974olsPVu2ZOs1M2slSfT399PX18fGjRvZsGHD6GmblqsOgb6+PoCDYzbupM+yZctiVKUSsX59RHd3RLbrbs2nuztbb6USZmYtV6lUoq+vL4Do6+uLSot3NvXWDwxEg/1qx54aioCPfhTuvz/7a76VTpzI1ivBJz/Z2nWbmY0eGQAtP00UNUcCzay3Y4PgL/4C7ruv9SEw6vhx2LwZfvZn4UMfKmYbZpauIsJgMiEAHRoEQ0PwkY8UFwKjTpyA9evhqqtg/vxit2Vm6WllGEw2BKBDg+CGG+DkyXK29corcP31sHNnOdszs7S0IgzOJASgA4Pg+HF49tnTnw0oyshIdkvqwICfMzCzYpxJGJxpCEAHBsH3vz/28wFFeOUV6O+Hz32u3O2aWTomEwatCAHowCA4erT8e/wrFdi2Ldu2h6Mws6JMJAxaFQLQgUEgtedhr3POgT17YNWq8rdtZuloJgxaGQJQ8JPFklZJek7SoKTb6iyXpHvz5fskXTreOif61HCrHD+eXSswMyvaWE8gtzoEXltpER9gOvBd4KeBGcA3gaU1ba4BHgMELAe+Pv56l7X0KeKJfN7znpY+/GdmNqZ6TwhP9olk2vRk8eXAYEQcAJC0FVgNfLuqzWrgwbzIPZJmS7ogIl4ssK5Je/GsrMrMpqra00Sjp4padiSQK/LU0HxeP8jRUD5vom2QtFbSgKQBGG55oc0q69kFM7NR1WEwqtWjlhYZBPWqrL3M20wbImJzRPRGRC/0tKS4yTjnnLZt2swSFfHa+wRe0+pRS4sMgiFgYdX0AuDQJNqcNS5o+FoHM7PWGw2B0QvDlUqlkCGsi7xGsBdYLGkR8AJwLfCBmjbbgXX59YMrgJfO1usDXV2wcmW7qzCzVNSGwOjpoCJGLS0sCCJiRNI64HGyO4i2RMR+STfnyzcBO8juHBoETgA3jbfeadPacwvpuedmbzAzMytaoxCAYkYtLfSBsojYQbazr563qep7AB+e2DpbU9tEnTwJy5e3Z9tmlo6xQmBUq8Og454snj0bXnqp3KOCadNgzRoPL2FmxWomBEa1Mgw6Lgje8pbsr/Oi30VQbeZM+NjHytuemaVnIiEwqlVh0HFB0N2dnat/4olyhqLu6sq2t2xZ8dsyszRNJgRGtSIMOi4IAB56CJYsKScIZs6Ez362+O2YWZrOJARGnWkYqJUPJZRB0jDwTzD3zTD/wuwMfrOGmdgDaZUKvPA8HP7BBMs8m8wBjrS7iJK5z2mYKn1eCMwFDvP6kRbqaabPjdb31oiouwPsuCA4E5IGsqeT0+E+p8F9TkNRfS50GGozMzv7OQjMzBKXWhBsbncBbeA+p8F9TkMhfU7qGoGZmZ0utSMCMzOr4SAwM0vclAwCSaskPSdpUNJtdZZL0r358n2SLm1Hna3URJ+vz/u6T9LXJF3Sjjpbabw+V7W7TNKrkt5fZn1FaKbPklZK+oak/ZJ2lV1jqzXx//ZPSnpU0jfzPo87ivHZTNIWSYclfavB8tbvvxq9zLhTP2RDXn8X+GlgBvBNYGlNm2uAx8jekLYc+Hq76y6hzz8PnJd/vzqFPle1+wrZKLjvb3fdJfw7zyZ7L/iF+fTcdtddQp9/G/jj/HsP8ENgRrtrP4M+vwu4FPhWg+Ut339NxSOCy4HBiDgQEaeArcDqmjargQcjsweYLamT3z82bp8j4msR8aN8cg/Z2+A6WTP/zgC3AF8ge8qy0zXT5w8AD0fE8wAR0en9bqbPAbxR2XgKs8iCoIQBaIoREbvJ+tBIy/dfUzEI5vP6x6qH8nkTbdNJJtqfXyP7i6KTjdtnSfOBNcAmpoZm/p3fBpwnaaekpyTdWFp1xWimz58C3kH2mttngPUR0YbXV5Wm5fuvjhx0bhz1RlmqvUe2mTadpOn+SHo3WRD8QqEVFa+ZPm8Ebo2IV8/0VX5niWb63AUsA94L/ATwpKQ9EfGdoosrSDN9vgr4BvAe4D8AfyfpqxFxrODa2qXl+6+pGARDZIMujVpA9pfCRNt0kqb6I+li4H7g6ojo5IH0oLk+9wJb8xCYA1wjaSQivlhKha3X7P/bRyLiOHBc0m7gEqBTg6CZPt8E/FFkJ9AHJX0PWAL8/3JKLF3L919T8dTQXmCxpEWSZgDXAttr2mwHbsyvvi8HXoqIF8sutIXG7bOkC4GHgV/t4L8Oq43b54hYFBEXRcRFwP8FfrODQwCa+3/7EeCdkrokdQNXAM+WXGcrNdPn58mOgJB0PvB24ECpVZar5fuvKXdEEBEjktYBj5PdcbAlIvZLujlfvonsDpJrgEHgBNlfFB2ryT7fDrwZ+Ez+F/JIdPDIjU32eUppps8R8aykLwH7gApwf0TUvQ2xEzT573wX8ICkZ8hOm9waER07PLWkzwMrgTmShoA7gDdAcfsvDzFhZpa4qXhqyMzMJsBBYGaWOAeBmVniHARmZolzEJiZJc5BYMmS9Dv5aJX78tE6r2jy9+6U9Iv59778fv2x2v++pI83sd5lkp7JR5W8V1PkcWg7+zkILEmSrgR+Gbg0Ii4GfpHXj9/SUETcHhF/n0/2AWMGwQT8GbAWWJx/VrVovWZjchBYqi4gG4rhJEBEHImIQ5Iul/QwgKTVkv5F0gxJMyUdyOc/IOn9kj4CzAP+QdI/5MtWSXo6Hxv/y1XbW5oPBHcg/73XyUePfFNEPJkPlfAg8L4i/wOYjXIQWKr+Flgo6TuSPiNpRT7/aeA/5d/fCXwLuIxsqIavV68gIu4lG+Pl3RHxbkk9wH3Ar0TEJcB/r2q+hGxwtMuBOyS9oaae+WRjyIzq9BFxrYM4CCxJEfEy2Sida4Fh4P9I+mBEjJANXPYOsp12P9mLQt4JfHWc1S4HdkfE9/JtVI8p/zcRcTIf+uAwcH7N7061EXGtgzgILFkR8WpE7IyIO4B1wK/ki75K9ha3fwX+nmzI7l8Ado+zStF4532y6vurnD7O1xCvf1lQp4+Iax3EQWBJkvR2SYurZv0c8E/5991kF4GfjIhhssH6lgD766zqx8Ab8+9PAiskLcq38VPN1pOPHvljScvzu4VuJBtJ1KxwU270UbMmzQL+VNJsstcaDpKdJoLsWsD5/PsRwD7gcNQfoXEz8JikF/PrBGuBhyVNIzsF9EsTqOl/AQ+QvVDmMTr/LXLWITz6qJlZ4nxqyMwscQ4CM7PEOQjMzBLnIDAzS5yDwMwscQ4CM7PEOQjMzBL3b1viHRyfxX3WAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "X = np.array([[0, 0], [1, 0], [0, 1], [1, 1]])\n",
    "y = (X[:,0] + X[:,1] == 1).astype(int)\n",
    "\n",
    "for i in [0, 1]:\n",
    "    plt.scatter(X[y == i][:,0], X[y == i][:,1], \n",
    "                marker=['o', 'x'][i], color=['b', 'k'][i],\n",
    "                s=1000)\n",
    "plt.xlabel('Switch 0')\n",
    "plt.ylabel('Switch 1')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "KNN will fail to properly classify the data. Furthermore, there does not exist a linear separation between the labeled classes. We cannot draw a linear boundary without cutting through a diagonal that connects two identically classified points. Consequently, training a linear classifier is also out of the question. What should we do? One approach is to define two nested if/else statements as our prediction model.\n",
    "\n",
    "**Listing 22. 2. Classifying data using nested if/else statements**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify(features):\n",
    "    switch0, switch1 = features\n",
    "    \n",
    "    if switch0 == 0:\n",
    "        if switch1 == 0:\n",
    "            prediction = 0\n",
    "        else:\n",
    "            prediction = 1\n",
    "    else:\n",
    "        if switch1 == 0:\n",
    "            prediction = 1\n",
    "        else:\n",
    "            prediction = 0\n",
    "    \n",
    "    return prediction\n",
    "\n",
    "for i in range(X.shape[0]):\n",
    "    assert classify(X[i]) == y[i]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to find a way to automatically derive accurate if / else statements from the training data.  We’ll start with a simple training example. Our training set will represent a series of recorded observations between a single light switch and a single bulb. Whenever the switch is on, the bulb is on, and vice-versa. \n",
    "\n",
    "**Listing 22. 3. Generating a single switch training set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "features: [[1]\n",
      " [1]\n",
      " [1]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [1]\n",
      " [0]]\n",
      "\n",
      "labels: [1 1 1 1 0 1 0 1 1 0]\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "y_simple = np.random.binomial(1, 0.5, size=10)\n",
    "X_simple = np.array([[e] for e in y_simple])\n",
    "print(f\"features: {X_simple}\")\n",
    "print(f\"\\nlabels: {y_simple}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll count all observations in which the switch is off and the light is off.\n",
    "\n",
    "**Listing 22. 4. Counting the off-state co-occurrences**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In 3 instances, both the switch and the light are off\n"
     ]
    }
   ],
   "source": [
    "count = (X_simple[:,0][y_simple == 0] == 0).sum()\n",
    "print(f\"In {count} instances, both the switch and the light are off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets count the instances in which both the switch and light bulb are turned on.\n",
    "\n",
    "**Listing 22. 5. Counting the on-state co-occurrences**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In 7 instances, both the switch and the light are on\n"
     ]
    }
   ],
   "source": [
    "count = (X_simple[:,0][y_simple == 1] == 1).sum()\n",
    "print(f\"In {count} instances, both the switch and the light are on\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These co-occurrences will prove useful during classifier training. Lets track the counts more systematically within a co-occurrence matrix `M`.\n",
    "\n",
    "**Listing 22. 6. Computing a co-occurrence matrix**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 0]\n",
      " [0 7]]\n"
     ]
    }
   ],
   "source": [
    "def get_co_occurrence(X, y, col=0):\n",
    "    co_occurrence = []\n",
    "    for i in [0, 1]:\n",
    "        counts = [(X[:,col][y == i] == j).sum()\n",
    "                  for j in [0, 1]] \n",
    "        co_occurrence.append(counts)\n",
    "    \n",
    "    return np.array(co_occurrence)\n",
    "\n",
    "M = get_co_occurrence(X_simple, y_simple)\n",
    "assert M[0][0] == 3\n",
    "assert M[1][1] == 7\n",
    "print(M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose that there's a flaw in the switch toggle. We flip the switch off, but the bulb stays on! Lets add this anomalous observation to our data, and subsequently recompute matrix `M`.\n",
    "\n",
    "**Listing 22. 7. Adding a flawed mismatch to the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 1]\n",
      " [0 7]]\n"
     ]
    }
   ],
   "source": [
    "X_simple = np.vstack([X_simple, [1]])\n",
    "y_simple = np.hstack([y_simple, [0]])\n",
    "M = get_co_occurrence(X_simple, y_simple)\n",
    "print(M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How accurately can we predict the state of the light-bulb if we know that the switch is off? In order to find out, we must divide `M[0]` by `M[0].sum()`. This will produce a probability distribution over possible light-bulb states whenever the switch-state is set to `0`.\n",
    "\n",
    "**Listing 22. 8. Computing bulb probabilities when the switch is off**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When the switch is set to 0, the bulb state probabilities are:\n",
      "[0.75 0.25]\n",
      "\n",
      "There is a 75% chance that the bulb is off.\n",
      "There is a 25% chance that the bulb is on.\n"
     ]
    }
   ],
   "source": [
    "bulb_probs = M[0] / M[0].sum()\n",
    "print(\"When the switch is set to 0, the bulb state probabilities are:\")\n",
    "print(bulb_probs)\n",
    "\n",
    "prob_on, prob_off = bulb_probs\n",
    "print(f\"\\nThere is a {100 * prob_on:.0f}% chance that the bulb is off.\")\n",
    "print(f\"There is a {100 * prob_off:.0f}% chance that the bulb is on.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the switch is off, we can predict the state of the bulb with 75% accuracy. Now lets optimize the accuracy for the scenario in which the switch is on. \n",
    "\n",
    "**Listing 22. 9. Predicting the state of the bulb when the switch is on**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When the switch is set to 1, the bulb state probabilities are:\n",
      "[0. 1.]\n",
      "\n",
      "We assume the bulb is on with 100% accuracy\n"
     ]
    }
   ],
   "source": [
    "bulb_probs = M[1] / M[1].sum()\n",
    "print(\"When the switch is set to 1, the bulb state probabilities are:\")\n",
    "print(bulb_probs)\n",
    "\n",
    "prediction = ['off', 'on'][bulb_probs.argmax()]\n",
    "accuracy = bulb_probs.max()\n",
    "print(f\"\\nWe assume the bulb is {prediction} with \"\n",
    "      f\"{100 * accuracy:.0f}% accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do we combine our two accuracy values into a single accuracy score? Naively, we could simply average 0.75 and 1.0. However, that approach would be erroneous. The two accuracies should not be weighted evenly, since the switch is on nearly twice as often as it is off.\n",
    "\n",
    "**Listing 22. 10. Counting the on / off states of the switch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The switch is off in 4 observations.\n",
      "The switch is on in 7 observations.\n"
     ]
    }
   ],
   "source": [
    "for i, count in enumerate(M.sum(axis=1)):\n",
    "    state = ['off', 'on'][i]\n",
    "    print(f\"The switch is {state} in {count} observations.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The switch is on more frequently than it is off. Hence, to get a meaningful accuracy score, we need to take the weighted average of 0.75 and 1.0.\n",
    "\n",
    "**Listing 22. 11. Computing total accuracy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our total accuracy is 91%\n"
     ]
    }
   ],
   "source": [
    "accuracies = [0.75, 1.0]\n",
    "total_accuracy = np.average(accuracies, weights=M.sum(axis=1))\n",
    "print(f\"Our total accuracy is {100 * total_accuracy:.0f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the switch is off, we predict that the bulb is off. Otherwise, we predict that the bulb is on. This model is 91% accurate. Furthermore, the model can be represented as a simple if/else Python statement, which we can train from scratch.\n",
    "\n",
    "**Listing 22. 12. Training a simple if/else model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if switch == 0:\n",
      "    prediction = 0\n",
      "else:\n",
      "    prediction = 1\n",
      "\n",
      "This statement is 91% accurate.\n"
     ]
    }
   ],
   "source": [
    "def train_if_else(X, y, feature_col=0, feature_name='feature'):\n",
    "    M = get_co_occurrence(X, y, col=feature_col)\n",
    "    probs0, probs1 = [M[i] / M[i].sum() for i in [0, 1]]\n",
    "    \n",
    "    if_else = f\"\"\"if {feature_name} == 0:\n",
    "    prediction = {probs0.argmax()}\n",
    "else:\n",
    "    prediction = {probs1.argmax()}\n",
    "    \"\"\".strip()\n",
    "\n",
    "    if probs0.argmax() == probs1.argmax():\n",
    "        if_else = f\"prediction = {probs0.argmax()}\"\n",
    "\n",
    "    accuracies = [probs0.max(), probs1.max()]\n",
    "    total_accuracy = np.average(accuracies, weights=M.sum(axis=1))\n",
    "    return if_else, total_accuracy\n",
    "\n",
    "if_else, accuracy = train_if_else(X_simple, y_simple, feature_name='switch')\n",
    "print(if_else)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're able to train a simple if/else model using a single  feature. Now, we'll figure out how to train a nested if/else model using two features.\n",
    "\n",
    "### 22.1.1 Training a Nested If/Else Model Using Two Features\n",
    "\n",
    "Lets return to our system of two light-switches connecting to a single staircase bulb. Our two features, `switch0` and `switch1`, correspond to columns `0` and `1` of matrix `X`. However, the `train_if_else` function can only train on one column at a time. Lets train two separate models and compare them.\n",
    "\n",
    "**Listing 22. 13. Training models on the two-switch system**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model trained on switch0 is 50% accurate.\n",
      "The model trained on switch1 is 50% accurate.\n"
     ]
    }
   ],
   "source": [
    "feature_names = [f\"switch{i}\" for i in range(2)]\n",
    "for i, name in enumerate(feature_names):\n",
    "    _, accuracy = train_if_else(X, y, feature_col=i, feature_name=name)\n",
    "    print(f\"The model trained on {name} is {100 * accuracy:.0f}% \"\n",
    "           \"accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " A single if/else statement is insufficient to capture the complexity of the problem. We must break the problem into parts by training two separate models; _A_ and _B_.  _Model A_ will consider only those scenarios in which `switch0` is set to off. We'll start by isolating a training subset that satisfies this boolean requirement.\n",
    " \n",
    " \n",
    "**Listing 22. 14. Isolating a training subset where `switch0` is off**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature matrix when switch0 is off:\n",
      "[[0 0]\n",
      " [0 1]]\n",
      "\n",
      "Class labels when switch0 is off:\n",
      "[0 1]\n"
     ]
    }
   ],
   "source": [
    "is_off = X[:,0] == 0\n",
    "X_switch0_off = X[is_off]\n",
    "y_switch0_off = y[is_off]\n",
    "print(f\"Feature matrix when switch0 is off:\\n{X_switch0_off}\")\n",
    "print(f\"\\nClass labels when switch0 is off:\\n{y_switch0_off}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`X_switch0_off[:,0]` always equals zero. Thus this zero column is now redundant. Lets delete it.\n",
    "\n",
    "**Listing 22. 15. Deleting a redundant feature column**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0]\n",
      " [1]]\n"
     ]
    }
   ],
   "source": [
    "X_switch0_off = np.delete(X_switch0_off, 0, axis=1) \n",
    "print(X_switch0_off)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll train an if/else model on the training subset. \n",
    "\n",
    "**Listing 22. 16. Training a model when `switch0` is off**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If switch 0 is off, then the following if/else model is 100% accurate.\n",
      "\n",
      "if switch1 == 0:\n",
      "    prediction = 0\n",
      "else:\n",
      "    prediction = 1\n"
     ]
    }
   ],
   "source": [
    "results = train_if_else(X_switch0_off, y_switch0_off, \n",
    "                        feature_name='switch1')\n",
    "switch0_off_model, off_accuracy = results\n",
    "print(\"If switch 0 is off, then the following if/else model is \"\n",
    "      f\"{100 * off_accuracy:.0f}% accurate.\\n\\n{switch0_off_model}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets train a corresponding model to cover all the cases where `switch0` is on. We'll start by filtering our training data based on the condition `X[:,0] == 1`.\n",
    "\n",
    "**Listing 22. 17. Isolating a training subset where `switch0` is on**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_X_y(X, y, feature_col=0, condition=0):\n",
    "    inclusion_criteria = X[:,feature_col] == condition\n",
    "    y_filtered = y[inclusion_criteria]\n",
    "    X_filtered = X[inclusion_criteria]\n",
    "    X_filtered = np.delete(X_filtered, feature_col, axis=1)\n",
    "    return X_filtered, y_filtered\n",
    "\n",
    "X_switch0_on, y_switch0_on = filter_X_y(X, y, condition=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll train `switch0_on_model` using the filtered training set.\n",
    "\n",
    "**Listing 22. 18. Training a model when `switch0` is on**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If switch 0 is on, then the following if/else model is 100% accurate.\n",
      "\n",
      "if switch1 == 0:\n",
      "    prediction = 1\n",
      "else:\n",
      "    prediction = 0\n"
     ]
    }
   ],
   "source": [
    "results = train_if_else(X_switch0_on, y_switch0_on, \n",
    "                        feature_name='switch1')\n",
    "switch0_on_model, on_accuracy = results\n",
    "print(\"If switch 0 is on, then the following if/else model is \"\n",
    "      f\"{100 * on_accuracy:.0f}% accurate.\\n\\n{switch0_on_model}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Together, our two models can be easily combined into a single nested if/else statement.\n",
    "\n",
    "**Listing 22. 19. Combining separate if/else models together**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if switch0 == 0:\n",
      "    if switch1 == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n",
      "else:\n",
      "    if switch1 == 0:\n",
      "        prediction = 1\n",
      "    else:\n",
      "        prediction = 0\n"
     ]
    }
   ],
   "source": [
    "def combine_if_else(if_else_a, if_else_b, feature_name='feature'):\n",
    "    return  f\"\"\"\n",
    "if {feature_name} == 0:\n",
    "{add_indent(if_else_a)}\n",
    "else:\n",
    "{add_indent(if_else_b)}\n",
    "\"\"\".strip()\n",
    "\n",
    "def add_indent(if_else):\n",
    "    return '\\n'.join([4 * ' ' + line for line in if_else.split('\\n')])\n",
    "\n",
    "nested_model = combine_if_else(switch0_off_model, switch0_on_model, \n",
    "                               feature_name='switch0')\n",
    "print(nested_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our combined model is 100% accurate. We can confirm by taking the weighted average of `off_accuracy` and `on_accuracy`. \n",
    "\n",
    "**Listing 22. 20. Computing total nested accuracy**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our total accuracy is 100%\n"
     ]
    }
   ],
   "source": [
    "accuracies = [off_accuracy, on_accuracy]\n",
    "weights = [y_switch0_off.size, y_switch0_on.size]\n",
    "total_accuracy = np.average(accuracies, weights=weights)\n",
    "print(f\"Our total accuracy is {100 * total_accuracy:.0f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're able to generate a nested two-feature model in an automated manner. Our strategy hinges on the creation of separate training sets. That separation is determined by the on/off states of one the features. This type of separation is called a **binary split**.\n",
    "\n",
    "**Listing 22. 21. Defining a binary split function**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split(X, y, feature_col=0, condition=0):\n",
    "    has_condition = X[:,feature_col] == condition\n",
    "    X_a, y_a = [e[has_condition] for e in [X, y]] \n",
    "    X_b, y_b = [e[~has_condition] for e in [X, y]]\n",
    "    X_a, X_b = [np.delete(e, feature_col, axis=1) for e in [X_a, X_b]]\n",
    "    return [X_a, X_b, y_a, y_b]\n",
    "\n",
    "X_a, X_b, y_a, y_b = split(X, y)\n",
    "assert np.array_equal(X_a, X_switch0_off) \n",
    "assert np.array_equal(X_b, X_switch0_on)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets define a `train_nested_if_else` to systematically execute our two-feature training logic from the previous code listings.\n",
    "\n",
    "**Listing 22. 22. Training a nested if/else model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if switch0 == 0:\n",
      "    if switch1 == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n",
      "else:\n",
      "    if switch1 == 0:\n",
      "        prediction = 1\n",
      "    else:\n",
      "        prediction = 0\n",
      "\n",
      "This statement is 100% accurate.\n"
     ]
    }
   ],
   "source": [
    "def train_nested_if_else(X, y, split_col=0, \n",
    "                         feature_names=['feature1', 'feature1']):\n",
    "    split_name = feature_names[split_col]\n",
    "    simple_model, simple_accuracy = train_if_else(X, y, split_col, \n",
    "                                                  split_name)\n",
    "    if simple_accuracy == 1.0:\n",
    "        return (simple_model, simple_accuracy)\n",
    "    \n",
    "    X_a, X_b, y_a, y_b = split(X, y, feature_col=split_col)\n",
    "    in_name = feature_names[1 - split_col] \n",
    "    if_else_a, accuracy_a = train_if_else(X_a, y_a, feature_name=in_name) \n",
    "    if_else_b, accuracy_b = train_if_else(X_b, y_b, feature_name=in_name)\n",
    "    nested_model = combine_if_else(if_else_a, if_else_b, split_name)\n",
    "    accuracies = [accuracy_a, accuracy_b]\n",
    "    nested_accuracy = np.average(accuracies, weights=[y_a.size, y_b.size])\n",
    "    if nested_accuracy > simple_accuracy:\n",
    "        return (nested_model, nested_accuracy)\n",
    "    \n",
    "    return (simple_model, simple_accuracy)\n",
    "\n",
    "feature_names = ['switch0', 'switch1']\n",
    "model, accuracy = train_nested_if_else(X, y, feature_names=feature_names)\n",
    "print(model)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our function trained a model that's 100% accurate. Given our current training set, that accuracy should hold even if we split on `switch1` instead of `switch0`.\n",
    "\n",
    "**Listing 22. 23. Splitting on `switch1` instead of `switch0`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if switch1 == 0:\n",
      "    if switch0 == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n",
      "else:\n",
      "    if switch0 == 0:\n",
      "        prediction = 1\n",
      "    else:\n",
      "        prediction = 0\n",
      "\n",
      "This statement is 100% accurate.\n"
     ]
    }
   ],
   "source": [
    "model, accuracy = train_nested_if_else(X, y, split_col=1,\n",
    "                                       feature_names=feature_names)\n",
    "print(model)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting on either feature will yield the same result. This is true of our two-switch system. However, this is not the case for many real-world training sets.\n",
    "\n",
    "### 22.1.2 Deciding which Feature to Split On"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose we wish to train an if/else model that predicts whether or not it’s raining outside. The model returns `1` if it is raining and `0` otherwise. We'll simulate the relationship between features (seasons and wetness) and rain-observation using random sampling.\n",
    "\n",
    "**Listing 22. 24. Simulating a rainy-day training set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "y_rain = np.random.binomial(1, 0.6, size=100) \n",
    "is_wet = [e if np.random.binomial(1, 0.95) else 1 - e for e in y_rain] \n",
    "is_fall = [e if np.random.binomial(1, 0.6) else 1 - e for e in y_rain]\n",
    "X_rain = np.array([is_fall, is_wet]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets train a model by splitting on the _Autumn_ feature.\n",
    "\n",
    "**Listing 22. 25. Training a model with an _Autumn_ split**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if is_autumn == 0:\n",
      "    if is_wet == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n",
      "else:\n",
      "    if is_wet == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n",
      "\n",
      "This statement is 95% accurate.\n"
     ]
    }
   ],
   "source": [
    "feature_names = ['is_autumn', 'is_wet']\n",
    "model, accuracy = train_nested_if_else(X_rain, y_rain, \n",
    "                                       feature_names=feature_names)\n",
    "print(model)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We trained a nested model that is 95% accurate. What if we split on the _Wetness_ feature instead?\n",
    "\n",
    "**Listing 22. 26. Training a model with a Wetness split**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if is_wet == 0:\n",
      "    prediction = 0\n",
      "else:\n",
      "    prediction = 1\n",
      "\n",
      "This statement is 95% accurate.\n"
     ]
    }
   ],
   "source": [
    "model, accuracy = train_nested_if_else(X_rain, y_rain, split_col=1,\n",
    "                                       feature_names=feature_names)\n",
    "print(model)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting on some features leads to more preferable results. How should we select the best feature for the split? \n",
    "One solution requires us to examine the distribution of classes within the training set. We'll do this by first printing the probability of rain.\n",
    "\n",
    "**Listing 22. 27. Computing the probability of rain**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It rains in 61% of our observations.\n"
     ]
    }
   ],
   "source": [
    "prob_rain = y_rain.sum() / y_rain.size\n",
    "print(f\"It rains in {100 * prob_rain:.0f}% of our observations.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It rains in 61% of total observations. How is that probability altered when we split on _Autumn_?\n",
    "\n",
    "**Listing 22. 28. Computing the probability of rain based on the season**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It rains 55% of the time when it is not autumn\n",
      "It rains 66% of the time when it is autumn\n"
     ]
    }
   ],
   "source": [
    "y_fall_a, y_fall_b = split(X_rain, y_rain, feature_col=0)[-2:]\n",
    "for i, y_fall in enumerate([y_fall_a, y_fall_b]):\n",
    "    prob_rain = y_fall.sum() / y_fall.size\n",
    "    state = ['not autumn', 'autumn'][i]\n",
    "    print(f\"It rains {100 * prob_rain:.0f}% of the time when it is \"\n",
    "          f\"{state}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we know that it is autumn, then we are slightly more confident in rain. Still, our confidence increase is not very substantial relative to the original training set. What if instead we were to split on _Wetness_ ? \n",
    "\n",
    "**Listing 22. 29. Computing the probability of rain based on wetness**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It rains 10% of the time when it is not wet\n",
      "It rains 98% of the time when it is wet\n"
     ]
    }
   ],
   "source": [
    "y_wet_a, y_wet_b = split(X_rain, y_rain, feature_col=1)[-2:]\n",
    "for i, y_wet in enumerate([y_wet_a, y_wet_b]):\n",
    "    prob_rain = y_wet.sum() / y_wet.size\n",
    "    state = ['not wet', 'wet'][i]\n",
    "    print(f\"It rains {100 * prob_rain:.0f}% of the time when it is \"\n",
    "          f\"{state}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The _Wetness_ split returns two class-label arrays whose corresponding rain probabilities are either very low or very high. These extreme probabilities are an indication of class imbalance.  When choosing between two splits, we should select the split that yields more imbalanced class-labels. Lets figure out how to quantify class imbalance. Generally, imbalance is associated with the shape of the class probability distribution. We can treat this distribution as a vector `v`, where `v[i]` equals probability of observing _Class i_. Lets visualize the distribution vectors associated with the _Autumn_ and _Wetness_ splits.\n",
    "\n",
    "**Listing 22. 30. Plotting the class distribution vectors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1h0lEQVR4nO3deXhU9fX48fdJQggSwhbWICZqkCWEsG/WgvwUsahfC+KCIlpKbV2qxbV92lr7rdWvVARxF6VWQRQR0NJWKyBoUCEUJSEsYQ+LECAsgYQs5/fHTNIQJsmEzJ2ZzJzX88wzM3c9E5g593Pv556PqCrGGGPCV0SgAzDGGBNYlgiMMSbMWSIwxpgwZ4nAGGPCnCUCY4wJc1GBDqCu4uPjNTExMdBhGGNMg5KRkZGnqm08zWtwiSAxMZE1a9YEOgxjjGlQRGRndfPs1JAxxoQ5SwTGGBPmHEsEIvKGiBwQkcxq5ouIzBCRHBH5TkT6OBWLMcaY6jl5jWA2MBN4q5r5o4Bk92Mg8JL72RjTwBUXF5Obm0thYWGgQwk7MTExdOrUiUaNGnm9jmOJQFVXiEhiDYtcB7ylrmJHX4lICxHpoKr7nIrJGOMfubm5NGvWjMTEREQk0OGEDVXl0KFD5ObmkpSU5PV6gbxGkADsrvQ+1z3tLCIyWUTWiMiagwcP+iU4Y8y5KywspHXr1pYE/ExEaN26dZ1bYoFMBJ7+h3gshaqqr6pqP1Xt16aNx26wxpggY0kgMM7l7x7IRJALnF/pfSdgb4BiMcaYsBXIRLAYmODuPTQIOGrXB4wxviIiTJkypeL91KlTefzxx2tcZ+HChWzYsKHGZXr16sXNN9/sVQzr1q1jyZIlXi0bSE52H50LrAIuEZFcEfmJiNwlIne5F1kCbANygNeAXzgVi3H5+99hwYJAR2GMfzRu3JgFCxaQl5fn9Tq1JYLs7GzKyspYsWIFBQUFtW4v7BOBqt6sqh1UtZGqdlLVWar6sqq+7J6vqnq3ql6kqj1V1epGOOz55+HppwMdhTH+ERUVxeTJk5k2bdpZ83bu3MmIESNITU1lxIgR7Nq1i/T0dBYvXsxDDz1EWloaW7duPWu9OXPmcNttt3HllVeyePHiiunDhg2rKH2Tl5dHYmIip0+f5ne/+x3z5s0jLS2NefPm8fjjjzN16tSK9VJSUtixYwc7duyga9euTJo0iZSUFMaPH8+///1vhg4dSnJyMt988w0Ajz/+OHfeeSfDhg3jwgsvZMaMGb75W/lkK6ZBKCmBOnQtNsZn/vOfYbUu07r1aDp3frBi+fbtJ9Khw0ROn84jK2vsGcv27r3cq/3efffdpKam8vDDD58x/Z577mHChAncfvvtvPHGG9x3330sXLiQa6+9ltGjRzN27FiP25s3bx6ffvopmzZtYubMmTWeIoqOjuaJJ55gzZo1zJw5E6DGU1M5OTm8//77vPrqq/Tv3585c+bwxRdfsHjxYp588kkWLlwIwMaNG1m2bBnHjx/nkksu4ec//3md7hnwxEpMhJHiYoiy1G/CSFxcHBMmTDjryHnVqlXccsstANx222188cUXtW5r9erVtGnThgsuuIARI0awdu1ajhw54rNYk5KS6NmzJxEREfTo0YMRI0YgIvTs2ZMdO3ZULPejH/2Ixo0bEx8fT9u2bfn+++/rvW/7WQgjJSVw3nmBjsKEI2+P4D0tHx0dX+f1K7v//vvp06cPd9xxR7XLeNPlcu7cuWzcuJHyMvjHjh3jgw8+YNKkSURFRVFWVgZQYx/+ystVXbZx48YVryMiIireR0REUFJS4nG5yMjIM+adK2sRhBFrEZhw1KpVK8aNG8esWbMqpg0ZMoR3330XgHfeeYdLL70UgGbNmnH8+PGztlFWVsb777/Pd999V3FOf9GiRcydOxdwlcfPyMgAYP78+RXrVd1eYmIia9euBWDt2rVs377dx5/23FgiCCMffACvvRboKIzxvylTppzRe2jGjBm8+eabpKam8re//Y3p06cDcNNNN/HMM8/Qu3fvMy4Wr1ixgoSEBBIS/lv84LLLLmPDhg3s27ePBx98kJdeeokhQ4acsZ/hw4ezYcOGiovFY8aM4fDhw6SlpfHSSy/RpUsXP3z62omr1E/D0a9fP7WBaYwJbtnZ2XTr1i3QYYQtT39/EclQ1X6elrcWQRh5/XX4+ONAR2GMCTaWCMLI00/DnDmBjsIYE2wsEYSR4mK7j8AYczZLBGGkpMR6DRljzmaJIIxYi8AY44klgjBiJSaMMZ5YIggj2dnwxz8GOgpj/OfDDz9ERNi4cWOtyz733HOcPHnSD1EFH0sEYaRtW2jRItBRGOM/c+fO5dJLL624i7gmlghMWPj972Hp0kBHYYx/nDhxgi+//JJZs2ZVJILly5czevToimXuueceZs+ezYwZM9i7dy/Dhw9n+PDhAMTGxlYsN3/+fCZOnAjAxIkT+fnPf87w4cO58MIL+fzzz7nzzjvp1q1bxTLl6//mN7+hV69eDBo0yCfF4ZxiiSBMqMITT8Dnnwc6EhOOhs0exux1swEoLi1m2OxhvP3d2wCcLD7JsNnDmJc5D4CjhUcZNnsYC7Jdoyjlncxj2OxhfLTpIwD2n9jv1T4XLlzIVVddRZcuXWjVqlVFjR9P7rvvPjp27MiyZctYtmxZrds+cuQIS5cuZdq0aVxzzTU88MADZGVlsX79etatWwdAQUEBgwYN4ttvv+Wyyy7jtSCu72KJIEyUlrqerfuoCRdz587lpptuAlw1hMoLxPnCNddcU1Eiul27dmeUjy4vGR0dHV3R+ujbt+8ZpaSDjf0shInySrXWa8gEwvKJyyteN4psdMb78xqdd8b75jHNz3gff178Ge/bx7avdX+HDh1i6dKlZGZmIiKUlpYiIlx77bXVloGuqnJp6qrLVS4RXbV8dHlZ6EaNGlVsw1flop1iLYIwUVzserYWgQkH8+fPZ8KECezcuZMdO3awe/dukpKSANiwYQNFRUUcPXqUzz77rGKdqiWj27VrVzFG8Ycffuj3z+BPlgjChLUITDiZO3cu119//RnTxowZw5w5cxg3bhypqamMHz+e3r17V8yfPHkyo0aNqrhY/NRTTzF69Gguv/xyOnTo4Nf4/c3KUIcJVTh1ytUiiI4OdDQm1FkZ6sCqaxlqO1EQJkRsmEpjjGd2aihMHD4M998Pq1cHOhJjTLCxRBAmDh+G6dPBizvtjTFhxhJBmCi/WGy9howxVVkiCBPl3Uet15AxpipLBGHCWgTGmOpYIggTpaUQEWEtAhM+RIQpU6ZUvJ86dSqPP/54jessXLiQDRs2nDU9Pz+f1q1bU97dftWqVYgIubm5ABw9epRWrVqdcddyZTt27GBOEA8YbokgTPTr50oGV18d6EiM8Y/GjRuzYMEC8vLyvF6nukTQokUL2rdvT3Z2NgDp6en07t2b9PR0AL766isGDhxIRITnn1RLBCaoVCqfYkxIi4qKYvLkyUybNu2seTt37mTEiBGkpqYyYsQIdu3aRXp6OosXL+ahhx4iLS2NrVu3nrHO0KFDK37409PTeeCBB854P2TIEEpLS3nooYfo378/qampvPLKKwA8+uijrFy5krS0NI/xBJqdMQ4TGzbAs8/CI49AcnKgozHh5P77768ozewraWlpPPfcc7Uud/fdd5OamsrDDz98xvR77rmHCRMmcPvtt/PGG29w3333sXDhQq699lpGjx7N2LFjz9rWkCFDWLFiBZMmTWLbtm3ccMMNFT/06enpPPbYY8yaNYvmzZuzevVqioqKGDp0KFdeeSVPPfUUU6dO5eOPP/bJ5/c1R1sEInKViGwSkRwRedTD/OYi8pGIfCsiWSJyh5PxhLPdu2HWLKhDK9mYBi8uLo4JEyYwY8aMM6avWrWKW265BYDbbruNL774otZtlbcItm/fTmJiIjExMagqJ06cICMjgwEDBvDJJ5/w1ltvkZaWxsCBAzl06BBbtmxx5LP5kmMtAhGJBF4ArgBygdUislhVK5+AuxvYoKrXiEgbYJOIvKOqp52KK1xZryETKN4cuTvp/vvvp0+fPtxxR/XHmeLFOdPk5GSOHDnCRx99xODBgwHXOANvvvkmSUlJxMbGoqo8//zzjBw58ox1ly9fXq/P4DQnWwQDgBxV3eb+YX8XuK7KMgo0E9e/QixwGAjeot0NmN1HYMJVq1atGDduHLNmzaqYNmTIkIrhK9955x0uvfRS4OxS1FUNHjyY6dOnVySCwYMH89xzzzFkyBAARo4cyUsvvUSx+wu3efNmCgoKat1uoDmZCBKA3ZXe57qnVTYT6AbsBdYDv1TVs/pfichkEVkjImsOHjzoVLwhzVoEJpxNmTLljN5DM2bM4M033yQ1NZW//e1vTJ8+HXCNZPbMM8/Qu3fvsy4Wg+v00O7du+nXz1XEc/DgwWzbtq0iEUyaNInu3bvTp08fUlJS+NnPfkZJSQmpqalERUXRq1evoLxY7FgZahG5ARipqpPc728DBqjqvZWWGQsMBX4FXAR8CvRS1WPVbdfKUJ+bBQvgjjvgm2/gkksCHY0JdVaGOrDqWobayRZBLnB+pfedcB35V3YHsEBdcoDtQFcHYwpbP/4xHD1qScAYczYnE8FqIFlEkkQkGrgJWFxlmV3ACAARaQdcAmxzMCZjjDFVOJYIVLUEuAf4F5ANvKeqWSJyl4jc5V7sj8AQEVkPfAY8oqrWwdEBn3wCN94IR44EOhJjTLBx9NKhqi4BllSZ9nKl13uBK52Mwbhs2QLvvQfPPx/oSIwxwcZKTIQJ6z5qjKmOJYIwYd1HjTHVsUQQJqxFYMLJAw88cMYdzSNHjmTSpEkV76dMmcKzzz7rcd3Zs2ezd2/VDo6hzRJBmGjaFBISrEVgwsOQIUMqKoOWlZWRl5dHVlZWxfz09HSGDh3qcV1LBCZk3Xcf5OZaIjDhoXLJ6KysLFJSUmjWrBlHjhyhqKioYlyBH/7wh/Tt25eRI0eyb98+5s+fz5o1axg/fjxpaWmcOnUqkB/Db+xnwRjjuGHDzp42bhz84hdw8qTnAZMmTnQ98vKgalXo2mq4dezYkaioqIpxBgYPHsyePXtYtWoVzZs3p1u3bjzwwAMsWrSINm3aMG/ePH7zm9/wxhtvMHPmTKZOnVpRRiIcWCIIE6+/DkuWuEpNGBMOylsF6enp/OpXv2LPnj2kp6fTvHlzEhIS+OSTT7jiiisAKC0tpUOHDgGOOHAsEYSJrCz47LNAR2HCVU1H8OedV/P8+PjaWwCelF8nWL9+PSkpKZx//vn85S9/IS4ujssvv7yihWDsGkHYKC626wMmvAwdOpSPP/6YVq1aERkZSatWrcjPz2fVqlXceOONHDx4sCIRFBcXV1xMDvaS0U6wRBAmSkqs66gJLz179iQvL49BgwadMa158+a0bduW+fPn88gjj9CrVy/S0tIqLi5PnDiRu+66yy4Wm9BjLQITbiIjIzl27MyK9rNnz654nZaWxooVK85ab8yYMYwZM8bp8IKKtQjCRPv2VoLaGOOZHSOGiT/9KdARGGOClbUIjDGOcGr0Q1Ozc/m7WyIIE4895hqq0hh/iImJ4dChQ5YM/ExVOXToEDExMXVaz04NhYnMTAiz8ikmgDp16kRubi4HDx4MdChhJyYmhk6dOtVpHUsEYaKkxHoNGf9p1KgRSUlJgQ7DeMlODYWJ4mK7j8AY45klgjBhLQJjTHXspyFMXHKJJQJjjGe1/jSIyI+Bp4G2gLgfqqpxDsdmfOiVVwIdgTEmWHlzjPh/wDWqmu10MMYYY/zPm2sE31sSaPjGjIEpUwIdRWBtPbyVFTvPri1jTLjzpkWwRkTmAQuBovKJqmpDnDQgGzZAZGSgowisZ9Kf4e3v3ib3V7m0iGkR6HCMCRreJII44CRwZaVpClgiaECsDDVMGzmNW1NvtSRgTBW1JgJVtcIEISDcy1CrKk0aNeHSzpcGOhRjgk611whE5GH38/MiMqPqw38hGl8I5xZBxt4MBrw+gE15mwIdijFBqaZjxPILxGv8EYhx1pAh0L17oKMIjCOFRwDo0Cx8Byc3pibS0KoD9uvXT9essdxk6kZVEZFAh2FMwIhIhqr28zSv1u6jItJGRKaKyBIRWVr+8H2YxvjeP3P+SWlZqSUBY2rgzX0E7+A6TZQE/AHYAax2MCbjgG7d4OmnAx2Ff6XvTmfUO6OYvW52oEMxJqh5kwhaq+osoFhVP1fVO4FB3mxcRK4SkU0ikiMij1azzDARWSciWSLyeR1iN3WQkwNHjwY6Cv8a1GkQH4z7gPGp4wMdijFBzZsOhcXu530i8iNgL1DrqAciEgm8AFwB5AKrRWSxqm6otEwL4EXgKlXdJSJt6xi/8YJqePYaipAIftztx4EOw5ig502L4H9FpDkwBXgQeB2434v1BgA5qrpNVU8D7wLXVVnmFmCBqu4CUNUD3gZuvFda6noOl/sIVJXr513PvMx5gQ7FmAah1kSgqh+r6lFVzVTV4araFzjsxbYTgN2V3ue6p1XWBWgpIstFJENEJnjakIhMFpE1IrLGhr6ru5IS13O4tAgOnTrEgYIDnCw+GehQjGkQqj1GdJ/aGYfrx/ufqpopIqOBXwNNgN61bNtTN42qfVWjgL7ACPc2V4nIV6q6+YyVVF8FXgVX99Fa9ms8uP5615gE4SD+vHi+uOML9Kz/bsYYT2o6WTALOB/4BpghIjuBwcCjqrrQi23nutcv1wnX9YWqy+SpagFQICIrgF7AZozPxMTAgjCpDJV1IIsLWlxAbHQs4vFYxBhTVU2JoB+QqqplIhID5AEXq+p+L7e9GkgWkSRgD3ATrmsClS0CZopIFBANDASm1eUDGFOuTMsY+/5Y2se2Z9ntywIdjjENRk2J4LSqlgGoaqGIbK5DEkBVS0TkHuBfQCTwhqpmichd7vkvq2q2iPwT+A4oA15X1cxz/jTGo337oGdPeO45uPXWQEfjnAiJ4M3r3qS4tLj2hY0xFWpKBF1F5Dv3awEucr8vH6oytbaNq+oSYEmVaS9Xef8M8EydojZ1cvo0HDrkqkAa6gZ18uoWF2NMJTUlgm5+i8I4qjwBhHL30be+fYuNeRt5fNjjREdGBzocYxqUan8aVHWnPwMxzgmH7qPr9q/j6z1f0ygihD+kMQ4J4WNEUy4cWgTPjnyWopIiKy5nzDnw5s5i08A1bw633QaJiYGOxPdOFp8k91guAI2jGgc4GmMaJm/KUI8WEUsYDVjnzvDWW9DPYyXyhu3F1S9y8YyL2ZG/IyD7b2jjeRjjiTc/8DcBW0Tk/0TELiCboDK2+1j+POLPJLZI9Pu+i4r285//XMrx4+v8vm9jfMmbWkO34ionsRV4U0RWuWv/NHM8OuMTK1e67i5evjzQkfheYotEHhj8QED2XVx8kOLiQ3ZdwjR4Xp3yUdVjwAe4Koh2AK4H1orIvQ7GZnykuBiKiiAihE7w5Rfm89PFPw3YKSGA2NieDBiQRWxsr4DFYIwveHON4FoR+RBYCjQCBqjqKFw1gR50OD7jA6HYa+ibPd8wN3Mu+YX5ft/3nj0vk5PzAGVlJbhqMxrTsHnz0zAWmKaqKypPVNWTInKnM2EZXwrF+wiuvOhK9vxqD81jmvt1v/n5K8nJuZeWLa+0U0ImZHhzsmBf1SQgIk8DqOpnjkRlfCrUWgT7T7hKXvk7CRQW7iYraywxMRfSrds71howIcObRHCFh2mjfB2IcU5iIvziF9A2BAYC3X9iPxfNuIgZX8/w635LS0+Rmfk/lJWdIiVlIY0atfDr/o1xUk0D0/wc+AX/LTZXrhnwpdOBGd9JS4MXXgh0FL7RtFFTHhn6CFcnX+23faoqmzdP5sSJ/5CSsoimTa0XtQktNZ0smAP8A/gz8Gil6cdV1ZuhKk2QKC2FsjLXqaGGflq7WeNm/O6Hv/PrPnNzp/H992+TmPhH4uOv8eu+jfGHmk4NqaruAO4Gjld6ICKtnA/N+Mpf/wrR0bB7d+3LBrPpX01n5c6Vft3n4cOfsnXrQ8TH/5gLLvi1X/dtjL/U1iIYDWTgGmu48rGkAhc6GJfxoVC4WFxYUsizXz3LNV2u4QcX/MAv+1RVtm//NU2bdqdr179ilVZMqKqpDPVo93OS/8IxTgiF7qMxUTFsvHsjRaVFftuniJCa+k9KS08QFRXrt/0a4281XSzuU9OKqrrW9+EYJzT0FsHxouPERsfSpFETmjRq4vj+VJV9+16jffuJNGrUmkaNWju+T2MCqaafhr/UME+By30ci3FIQ28RTP54MvuO72PZ7cv8chNXfv5yNm/+GZGRTWnXbrzj+zMm0Go6NTTcn4EY5/TvD488Ao0baLn+UReP4sipI367k7dly+H07v0lcXGD/bI/YwJNqqunLiKXq+pSEfmxp/mqusDRyKrRr18/XbNmTSB2bUJcQcFGSkoO07z5kECHYozPiUiGqnoclaSmU0M/xFVozlPHaQUCkghM3RUUwOnT0LJloCOpm3X715F5IJObU24mMsLZcg7FxflkZl5LWdkpBg7MISKigTafjDkHNZ0a+r37+Q7/hWOc8Kc/wdSprmTQkMxaO4s5mXO49pJriWsc59h+VEvJzh5PYeF2evVaaknAhB1vylC3FpEZIrJWRDJEZLqIWDeKBqS4uGH2GJo+ajqrfrLK0SQAsH37bzl8eAkXX/w8LVr45x4FY4KJN3fIvAscBMbgKkl9EJjnZFDGt0pKGl6PodOlp4mQCLq07uLofg4ceI9du/5Mhw4/pWPHnzm6L2OClTeJoJWq/lFVt7sf/wu0cDgu40MNrUWwbPsyLppxEeu/X+/ofk6c+JaNG+8gLm4IyckzbXwBE7a8SQTLROQmEYlwP8YBf3c6MOM7Da1FEBsdS/+O/UlunezYPk6fzmP9+uuIimpBjx4fEBER7di+jAl2Nd1ZfJz/1hj6FfC2e1YEcAL4vePRGZ+47jro1oAqJ/dP6M+CG53rlKaqbNhwE6dP76d37xU0btzesX0Z0xBU2yJQ1WaqGud+jlDVKPcjQlWdvXpnfGrUKPjlLwMdRe1UldcyXuPE6ROO7kdE6NTpXrp2nUVc3ABH92VMQ+DVmWMRaQkkAzHl06oOX2mC14EDrjEJOnQIdCQ1W5W7iskfTyY6Mprb0253ZB+nT+cRHR1PfPx1jmzfmIbIm+6jk4AVwL+AP7ifH3c2LONL99wDI0YEOoraDTl/CN9M+obxqc7U9zl+PIOvvkrk4MEPHdm+MQ2VNxeLfwn0B3a66w/1xtWFtFYicpWIbBKRHBF5tIbl+otIqYiM9SpqUyclJcHfa6hMywDX9YGoCGeCbdIkmfbtJ9C8ud0rYExl3iSCQlUtBBCRxqq6EbiktpVEJBJ4AddA992Bm0WkezXLPY2rpWEcUFwc3L2GSspKGPj6QF5a/ZIj2y8rO01p6SmiouLo0uVFoqPjHdmPMQ2VN4kgV0RaAAuBT0VkEbDXi/UGADmquk1VT+O6Mc3Tidl7gQ+AA15FbOos2FsEx4uOc3Gri2kf60zvnS1b7uU///kBpaWFjmzfmIau1p8HVb3e/fJxEVkGNAf+6cW2E4DKo+TmAgMrLyAiCcD1uMY26F/dhkRkMjAZoHPnzl7s2lQW7DeUtWzSkrlj5jqy7T17Xmbfvlfp3PkxIiNjal/BmDDkba+hPsCluO4r+NJ9hF/rah6mVa15/RzwiKqW1nRXp6q+CrwKrjLU3sRs/uuee/47OE2wWblzJUktk+gU18nn287PX0lOzr20anU1SUl/9Pn2jQkVtSYCEfkdcAP/LTv9poi87y41UZNc4PxK7ztx9imlfsC77iQQD1wtIiWqutCL2I2X/ud/Ah2BZ2Vaxh2L7qBTXCeWT1zu020XFu4mK2ssMTEX0q3bO7guRRljPPGmRXAz0LvSBeOngLVAbYlgNZAsIknAHuAm4JbKC6hqUvlrEZkNfGxJwPdyclwXiy+4INCRnClCIvhswmccKzrm0+2Wlp4iM/N6yspOkZKynEaNWvh0+8aEGm8SwQ5cN5KVX2lrDGytbSVVLRGRe3D1BooE3lDVLBG5yz3/5XOK2NTZuHGQkAAffRToSM52QQvfZidVZfPmyZw4sZaUlEU0bdqAamsYEyA11Rp6Htc5/SIgS0Q+db+/AvjCm42r6hJgSZVpHhOAqk70LmRTV8FYdG7aqmms3b+WWdfOIjrSdwXfcnOn8f33b5OY+Efi4z0NrmeMqaqmFkH5wMAZQOVbMZc7Fo1xRDD2GiosKaTgdIFPk4Cqcvx4BvHxY7jggt/4bLvGhLqahqr8a/lrEYkGykcI2aSqxU4HZnwnGFsEj/3gMVR92wFMROjW7W1UT9vYAsbUgTe1hoYBW3DdJfwisFlELnM2LONLwdQiyC/MJ2NvBoDPfqxLSk6QlXUTp05tRURszGFj6sibO4v/Alypqj9U1cuAkcA0Z8MyvvTMMzBpUqCjcJnx9Qz6v9af7Ue2+2ybp05tJj9/KadO+W6bxoQTb44TG6nqpvI3qrpZRILsRIOpyQ03BDqC/7p/0P10je9KUsuk2hf2UrNmfRg4cBtRUbE+26Yx4cSbFkGGiMwSkWHux2u4LiCbBuLrr2HHjkBH4RLXOI5xPcb5ZFt5eYvZufNJVNWSgDH14E0iuAvIAu7DVZJ6g3uaaSCuuAKmTw9sDPtP7Gfk2yPJPJDpk+0VFGSTnX0rBw8uoKysyCfbNCZc1XhqSEQigAxVTQGe9U9IxteCoQz15kObyT6YTUxU/Qu/FRfnk5l5HRERTUhJ+dCKyRlTTzUmAlUtE5FvRaSzqu7yV1DGt4KhDPVlF1zG9l9uJzKifjV/VEvJzr6FwsLt9Oq1lJiY82tfyRhTI29+HjrgurP4G6CgfKKqXutYVMZnVAN/H8G6/evo1a5XvZMAwPbtv+Xw4X+QnPwiLVrYSGPG+II3ieAPjkdhHFNa6noOVItg+5Ht9H+tP08Me4LHfvBYvbZ14MB77Nr1Zzp0+CkdO9plKmN8paZaQzG4LgpfDKwHZqlqkFa1N9URgQ8+gG4Bqr12fvPzee2a17jyoivrtZ0TJ75l48Y7iIsbQnLyTLtz2Bgfkupu8xeReUAxsBLXuMM7VfWXfozNo379+umaNWtqX9CEDFVl7drBFBXtpm/fDBo3dmZIS2NCmYhkqGo/T/NqOmHQXVV7ujcwC/jGieCMs06fhqVLoXt38Pcon498+gh9O/at930DIkKPHu9TXJxnScAYB9R0H0FFYTk7JdRw5efDqFHw8cf+3W9hSSGfbvuUb/d/W6/tHDq0BNUyYmLOp1mz3j6KzhhTWU0tgl4iUj50lABN3O8FUFWNczw6U2/lYxX7+2JxTFQMayav4XSpN8Nbe3b06JesX/8junR5lY4df+rD6IwxldVUhtoGeQ0Bxe52nT+7j+47vo+WTVoSExVTrxvI4uKG0KPHfFq3vs6H0RljqvKmxIRpwALRIvjJ4p8w6PVB5zzeQFHRfgoKNiAitGkzhoiIIKmhbUyIsm9YiCtPBP5sETw45EG+P/H9OXXxLCsrIitrDIWF2xg4cBuRkU0ciNAYU5klghDXqRP861/Qs6f/9nl50uXnvO6WLfdx7Fg63bu/Z0nAGD+xU0MhrmlTuPJK6NDB+X2t3LmSJ1c+yaniU+e0/p49L7Nv36t07vxr2rYNokEUjAlxlghC3KFD8P77sH+/8/v6Z84/eWnNS+d0Sig/fyU5OffSqtXVJCU94UB0xpjqWCIIcZs2wbhx8G39uvN75U8j/sS3d31b555ChYW7ycoaS0zMhXTvPgcR67BmjD9ZIghx/ug1pKrkncwDoFWTVnVat7T0FJmZ11NWVkhKyiKiopo7EaIxpgaWCEKcP+4jWLxpMYnPJbJ239o6raeqbN48mRMn1tKt2zs0bdrVoQiNMTWxXkMhzh8tgm5tujGpzyRS26XWaT0RoUWLYTRtmkJ8/GiHojPG1MYSQYjzR4ugS+suPHfVc3Vap6ysiIiIxnTo8BNngjLGeM1ODYW4oUPhyy+dGY+gtKyU3y79LbnHcuu03qlTW/n664s5dOjvvg/KGFNnlghCXMuWMGQIxMb6fttr9q7h6S+f5uvcr+u0XmRkLLGxfTjvPLsmYEwwqHZgmmBlA9PUzdatsGIFjBkDcQ7Ui919dDcJcQlESO3HFKplqJZZ7SBjAqCmgWmsRRDivvwS7rwTDh707XYLThcArqEovUkCADt3/onvvhtJaelJ3wZjjKkXSwQhzomic0UlRfR4sQdPrnzS63Xy8hazY8fvaNy4IxERVkPImGDiaCIQkatEZJOI5IjIox7mjxeR79yPdBHp5WQ84ai815Avu48WlxVzY48bGdxpsFfLFxRkk519K7GxfenS5VUbeN6YIOPYyVpx1Ql4AbgCyAVWi8hiVd1QabHtwA9V9YiIjAJeBQY6FVM4cqJFEBsdy9NXPO3VssXF+WRmXkdERAwpKR9aRVFjgpCTLYIBQI6qblPV08C7wBlDTalquqoecb/9CujkYDxhydctgg+zPyRjb4ZXy6qWkp19C4WF2+nR4wNiYs73TRDGGJ9yMhEkALsrvc91T6vOT4B/eJohIpNFZI2IrDno66ueIe622+C773zTY6hMy3jss8f49dJfe7X89u2/5fDhf3DxxTNo0eIH9Q/AGOMIJ/vxeToR7LGvqogMx5UILvU0X1VfxXXaiH79+jWs/q4B1rq16+ELERLBNz/9hiOnjtS67IED77Fr15/p0OGndOx4l28CMMY4wskWQS5Q+VxAJ2Bv1YVEJBV4HbhOVQ85GE9Y+uoreP55qO/tIiVlJagqcY3juKDFBTUuq6p8//0c4uKGkJz8vF0cNibIOZkIVgPJIpIkItHATcDiyguISGdgAXCbqm52MJawtWQJ3Hdf/bfzh+V/4PK3LqeopKjWZUWEHj3m07PnR0RENK7/zo0xjnIsEahqCXAP8C8gG3hPVbNE5C4RKT9X8DugNfCiiKwTEbtl2MdKSlw9hup7UJ7UMomUNik0jqr+h72srIScnAcpKtpPREQUjRrVbWwCY0xgOHqvv6ouAZZUmfZypdeTgElOxhDuiot902Pozt53Qu+alyko+Ja9e1+iWbO+tGt3c/13aozxC7uzOMQVF9fvHoL9J/bzYfaHeFOTqlmzvgwcuMWSgDENjCWCEFdSUr8WwStrXuGG929gR/6Oapc5dmw1e/e+DkDjxh3PfWfGmICw6qMh7sgROH4cOnc+t/VLykr4OvdrhnYe6nF+UdF+MjL6ERHRiP79M4mMbFqPaI0xTqmp+qjVAw5xLVu6HudCVYmKiKo2CZSVFZGVNYaSkiP06bPKkoAxDZSdGgpxixbBiy/Wfb1tR7bR48Ue1Q46o6ps2XIvx46l07XrbGJj6zZesTEmeFgiCHHvvQfTptV9vSOnjtA8pjnnN/dcH2jv3pfZt+81Onf+NW3b3lDPKI0xgWSnhkLcuXYf7duxL6t+ssrjvPz8leTk3EerVleTlPREPSM0xgSatQhCXPkNZXWxZMsSCksKPc4rLNxNVtZYYmIupHv3ObiqjRtjGjJLBCGuri2CnMM5jJ4zmqnpUz3O37jxdsrKTpGSsoioqOY+itIYE0h2aijE1bVFcHGri/n3hH+T1j7N4/zk5JkUFe2hadOuvgnQGBNwlghC3PvvQ2lp3da5POnys6YdP76O2NheNG3anaZNu/soOmNMMLBTQyEuNhaae3kG59YFtzLzm5lnTT9+PIOMjL7s3fuKj6MzxgQDSwQh7oUX4I03al+usKSQo0VHKThdcNa82NjeXHzxc7Rrd6sDERpjAs1KTIS4gQOhVSv4h8dBQM+mqhUDyZSUnKCk5IiNNWyCTllZGWVlZUT5ajDuMGAlJsKYN72GNuVtolWTVrRp2qYiCagqGzdO5NixLxkwYDNRUc38EK0xcPLkSfbs2VPjY9++fbz33ntcf/31gQ43JFgiCHHe9Bqa/PFkvj/xPdl3Z1ckgp07/0Re3gdcdNFUSwLGJ8rKyjhw4AB79uxh79691f7I5+fnn7VuXFwcHTt2JCEhgeHDh5OQkMBFF13k/w8RoiwRhDhvWgSvjH6F3Ud3VySBvLyP2LHjt7RrdyudOv3KD1Gahs7bo/iSkpIz1ouMjKR9+/YkJCTQpUuXih/5yo+OHTvSrJkdjDjJEkGI86ZF0DW+K13jXfcFFBRkk509ntjYvnTp8qoNPB/m6nMU36xZs4ofc08/8AkJCbRr147ISLs7PdAsEYS4jRuhrMzzvI82fcTft/yd/7vi/4hrHEdxcT6ZmdcRERFDSsqHREY28W+wxq/sKN6Us0QQ4iIjXQ9PNuZt5ItdX3Beo/NQLSU7ezyFhdvp1Wup9RRqwOwo3tSVdR8NcQ8/DH37wo03ep5fXFpMo8hGbNv2a3bt+jPJyS+SkPBz/wZpvHauR/ERERF06NDB4w97+UXYhIQEO4oPYdZ9NIy9/jqcOnVmIigtK2XL4S10je9Ko0jXBYSoqOZ06PAzOna8K0CRhjc7ijeBZIkgxHm6WPzO+neYuHAiX0/6mn4d+yEidO78yBk3kxnf8cVRfPm5+MpH73YUb3zFEkGI89R99Orkq3nmimdIjU8kI6MfF130DC1bXm5JoI7sKN6ECksEIc5TiyD+vHimDJlCYeFORCKJjLQjyqp8dRQ/bNgwjz/ydhRvgoklghCmCtHR0Lix631RSRGTP57Mg4MfJKVtCjExF9Cnz9dh1RKwo3hjzmaJIISJQEGlYqJZB7P4ePPH/KhTOyLydtC162wiI88LXIA+5ouj+OTkZDuKN2HHEkEY6dOhD+vv/IDNmSMpjroUkehAh+QVO4o3xlmWCELYqVMweTKMHw8pQ3KJj45gx+bxxMR0pHv3eUREBP6fvz5H8eV3t9pRvDH1E/hfAuOYwkJ4+21ISSvk1u/SGN2xMXd2zqdPn1VER8c7um87ijem4bBEEMLKD6JjoiP5RdIlJGg6Xbu+R2xsar22a0fxxoQWRxOBiFwFTAcigddV9akq88U9/2rgJDBRVdc6GVM4KS52PRee/JLLY9Pp3Pkx2ra9odrlnTyK79ixI+3atbMRpYwJQo59K0UkEngBuALIBVaLyGJV3VBpsVFAsvsxEHjJ/Wx8oPyA/OudbzNs2EhKSiawfPlyO4o3xpzBycOzAUCOqm4DEJF3geuAyongOuAtdVW++0pEWohIB1Xd52BcYWPb9u0Q0ZRFs07w4Uv/ArqdMb+6o/jKZQzsKN6Y0OfkNzwB2F3pfS5nH+17WiYBOCMRiMhkYDJA586dfR5oqOqZEse1oyfSqm1Tulz4pB3FG2M8cjIReLpdtWrNa2+WQVVfBV4FVxnq+ocWHlq3bs2iRR8FOgxjTJCLcHDbuUDl0U06AXvPYRljjDEOcjIRrAaSRSRJXLew3gQsrrLMYmCCuAwCjtr1AWOM8S/HTg2paomI3AP8C1f30TdUNUtE7nLPfxlYgqvraA6u7qN3OBWPMcYYzxztDqKqS3D92Fee9nKl1wrc7WQMxhhjaubkqSFjjDENgCUCY4wJc5YIjDEmzFkiMMaYMCeu67UNh4gcBHYGOo5K4oG8QAdRi2CP0eKrn2CPD4I/xnCI7wJVbeNpRoNLBMFGRNaoar9Ax1GTYI/R4qufYI8Pgj/GcI/PTg0ZY0yYs0RgjDFhzhJB/b0a6AC8EOwxWnz1E+zxQfDHGNbx2TUCY4wJc9YiMMaYMGeJwBhjwpwlgjoSkVYi8qmIbHE/t/SwzPkiskxEskUkS0R+6Ye4rhKRTSKSIyKPepgvIjLDPf87EenjdEx1jG+8O67vRCRdRHr5Mz5vYqy0XH8RKRWRscEWn4gME5F17v93nwdTfCLSXEQ+EpFv3fH5tdqwiLwhIgdEJLOa+YH+jtQWn3PfEVW1Rx0ewP8Bj7pfPwo87WGZDkAf9+tmwGagu4MxRQJbgQuBaODbqvvDVe77H7hGhRsEfO3Hv5k38Q0BWrpfj/JnfN7GWGm5pbiq6o4NpviAFrjGBO/sft82yOL7dfn3BWgDHAai/RjjZUAfILOa+QH7jngZn2PfEWsR1N11wF/dr/8K/E/VBVR1n6qudb8+DmTjGovZKQOAHFXdpqqngXfdcVZ2HfCWunwFtBCRDg7GVKf4VDVdVY+4336Fa7Q6f/LmbwhwL/ABcMCfweFdfLcAC1R1F4Cq+jNGb+JToJmICBCLKxGU+CtAVV3h3md1AvkdqTU+J78jlgjqrp26R1FzP7etaWERSQR6A187GFMCsLvS+1zOTjzeLOOUuu77J7iOzPyp1hhFJAG4HngZ//Pmb9gFaCkiy0UkQ0Qm+C067+KbCXTDNRzteuCXqlrmn/C8EsjvSF359Dvi6MA0DZWI/Bto72HWb+q4nVhcR4/3q+oxX8RW3a48TKvaL9ibZZzi9b5FZDiu/+SXOhqRh117mFY1xueAR1S11HVQ61fexBcF9AVGAE2AVSLylapudjo4vItvJLAOuBy4CPhURFY6/N2oi0B+R7zmxHfEEoEHqvr/qpsnIt+LSAdV3eduNnpsfotII1xJ4B1VXeBQqOVygfMrve+E66irrss4xat9i0gq8DowSlUP+Sm2ct7E2A94150E4oGrRaREVRcGSXy5QJ6qFgAFIrIC6IXrGlUwxHcH8JS6TnLniMh2oCvwjR/i80YgvyNeceo7YqeG6m4xcLv79e3AoqoLuM+BzgKyVfVZP8S0GkgWkSQRiQZucsdZ2WJggrtnxCDgaPkprmCIT0Q6AwuA2/x0BFvnGFU1SVUTVTURmA/8wk9JwKv4cP1f/IGIRInIecBAXNengiW+XbhaK4hIO+ASYJuf4vNGIL8jtXL0O+LPq+Kh8ABaA58BW9zPrdzTOwJL3K8vxdWk/A5XU3gdcLXDcV2N68hvK/Ab97S7gLvcrwV4wT1/PdDPz3+32uJ7HThS6e+1JgD/tjXGWGXZ2fix15C38QEP4eo5lInrlGTQxOf+jnzi/v+XCdzq5/jmAvuAYlxH/z8Jsu9IbfE59h2xEhPGGBPm7NSQMcaEOUsExhgT5iwRGGNMmLNEYIwxYc4SgTHGhDlLBCZouCt6rhORTBF5390X3tt1J4rIzDru70Q1058Qkf/nfr1cRPq5Xy8RkRbuxy/quK9EEVERubfStJkiMrGW9SaKSMdq5s0Wke3uv9m3IjLCiziWiEiLusRuQp8lAhNMTqlqmqqmAKdx9aGuICKR/ghCVX+nqv/2MP1qVc3HVeWzTonA7QDwS/cNV96aiKv/fXUeUtU04H68qIFU6TMYU8ESgQlWK4GL3fX1l4nIHGC9iMSIyJsisl5E/uOuu1LufBH5p7hq4v++fKKILHQXYcsSkcmVdyIifxGRtSLymYi0cU+bLR7GGhCRHSISDzwFXOQ+En9GRP4mItdVWu4dEbnWw2c6iOsmxNurzhCRNBH5yl1r/kMRaemOoR/wjntfTWr4e62iUoG06j5z+Wdwt1CyReQ19zKf1LJ9E8IsEZigIyJRuOqtr3dPGoDrTtXuwN0AqtoTuBn4q4jEVFpuPJAG3FB+Sge4U1X74vpRvU9EWrunNwXWqmof4HOgInnU4lFgq7v18hCuOz7vcMfeHFfd+CXVrPsUMMVD6+YtXAXtUt2f+/eqOh9YA4x37+tUDTFdBSys9L66z1xZMvCCqvYA8oExNWzfhDBLBCaYNBGRdbh+/HbhqtcE8I2qbne/vhT4G4CqbgR24iq/DPCpqh5y/2Au4L/VGe8TkW9x1XA/H9cPIEAZMM/9+m3OsZqjqn6Oq/XSFldy+kBVPdbZd3+Ob3CNHQBUJI8W7u2Aa5yLy7zc/TMiss0d/5OVplf3mSvbrqrr3K8zgEQv92lCjFUfNcHklPt8dwVX/T4KKk+qYf2q9VJURIYB/w8YrKonRWQ5EINn9am38jdcrZGbgDtrWfZJXEXrVtRjf+UewpX07sOVQPrW4TMXVXpdiqt0tQlD1iIwDc0KXD+4iEgXoDOwyT3vCnGNKd0E18hxXwLNgSPuH8SuuIYgLBcBlF8LuAX4wssYjuMagrSy2bgu2KKqWTWt7G7JbABGu98fBY6IyA/ci9yG61RVdfuqur0yYDoQISIjqfkzG3MWaxGYhuZF4GURWY9rmMOJqlrkbjl8gevI/GJgjqqucS93l4h8hythfFVpWwVADxHJAI4CN3oTgKoeEpEvxTXI+D9U9SFV/V5EsjnzPH1N/gT8p9L7292f6zxcpZnLB3af7Z5+CtcRvsfrBKqqIvK/wMO4qoBW95mNOYtVHzXGB9w/4OuBPu4jfGMaDDs1ZEw9uW8+2wg8b0nANETWIjDGmDBnLQJjjAlzlgiMMSbMWSIwxpgwZ4nAGGPCnCUCY4wJc/8fBCewIudxvUEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_class_distribution(y):\n",
    "    prob_rain = y.sum() / y.size\n",
    "    return np.array([1 - prob_rain, prob_rain])\n",
    "\n",
    "def plot_vector(v, label, linestyle='-', color='b'):\n",
    "    plt.plot([0, v[0]], [0, v[1]], label=label, \n",
    "             linestyle=linestyle, c=color)\n",
    "\n",
    "classes = [y_fall_a, y_fall_b, y_wet_a, y_wet_b]\n",
    "distributions = [get_class_distribution(y) for y in classes]\n",
    "labels = ['Not Autumn', 'Autumn', 'Not Wet', 'Wet']\n",
    "colors = ['y', 'g', 'k', 'b']\n",
    "linestyles = ['-.', ':', '-', '--']\n",
    "for tup in zip(distributions, labels, colors, linestyles):\n",
    "    vector, label, color, linestyle = tup\n",
    "    plot_vector(vector, label, linestyle=linestyle, color=color)\n",
    "\n",
    "plt.legend()\n",
    "plt.xlabel('Probability Not Rain')\n",
    "plt.ylabel('Probability Rain')\n",
    "plt.axis('equal')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The balanced _Autumn_ vectors are much shorter than the vectors associated with _Wetness_. This isn't a coincidence. Imbalanced distributions are proven to have greater vector magnitudes. Also, the magnitude is equal to the square-root of `v @ v`. Hence, the dot-product of a distribution vector with itself will be greater if that vector is more imbalanced! Lets demonstrate this property for every 2D vector `v = [1 - p, p]`.\n",
    "\n",
    "**Listing 22. 31. Plotting the distribution vector magnitudes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEGCAYAAAB1iW6ZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABNdklEQVR4nO3dd3hUxfrA8e+kVyAk9JLQWxokdBSQIkVARaQroBQR2++KvWDnXvEqXAQElagooIAKgqggvScQWug1oaZDepvfHyeJAVI2YXfP7mY+z5OHZPeU92zCu7NzZt4RUkoURVEU62endwCKoiiKcaiEriiKYiNUQlcURbERKqEriqLYCJXQFUVRbISDXif28fGRfn5+ep1eURTFKkVERMRJKWsU95xuCd3Pz4/w8HC9Tq8oJYqOjgagQYMGOkeiKHcSQlwo6TndErqiWKqxY8cCsHnzZn0DUZRyUgldUW7zxhtv6B2ColSISuiKcpvevXvrHYKiVIhK6Eqh7OxsYmJiyMjI0DsUXeXk5ADg4FC5/3u4uLhQv359HB0d9Q5FMVDl/otVbhETE4Onpyd+fn4IIfQORzcnTpwAoEWLFjpHoh8pJfHx8cTExNCoUSO9w1EMVOY4dCHE10KI60KIIyU8L4QQc4QQp4UQh4QQ7YwfpmIOGRkZeHt7V+pkDlC3bl3q1q2rdxi6EkLg7e1d6T+tWRtDJhaFAf1Keb4/0Cz/axIw/+7DUvRS2ZM5gKenJ56ennqHoTv1t2B9ykzoUsqtQEIpmwwBvpWa3UA1IUQdYwV4u+s3MnhnzVGysnNNdQqlksvIyFAtU8VkZv91kqOXk01ybGNM/a8HRBf5OSb/sTsIISYJIcKFEOGxsbEVOtnR48d4cN8Yfl66oEL7K5ZNCFE4Dhy0G5Q1atTggQceMPq5FixYwLfffgtAWFgYly9fBuDChQtcuFDi3I1b+Pn5ERcXZ/TYFNv0+9ZdDN72AAd3/G6S4xsjoRf3uazYVTOklAullKFSytAaNYqduVqmnqGBNHK+Sc1Ty1lz8HKFjqFYLnd3d44cOUJ6ejoAf/31F/XqFds+uGtTpkzhscceA25N6PXq1TPZOZXKK+ryDc5v+AJfu1ge7d3VJOcwRkKPAYrOka4PmC7T2tnj3vFx7rU/zKcr/+bUtZsmO5Wij/79+7N27VoAli5dysiRIwuf27t3L126dKFt27Z06dKlcERKWloajz76KIGBgQwfPpyOHTsWlpbw8PDg9ddfJygoiE6dOnHt2jUAZsyYwaxZs1ixYgXh4eGMHj2a4OBg7O3t8ff3L2x5h4eH06NHDwDi4+Pp27cvbdu2ZfLkyRRd8WvJkiV06NCB4OBgJk+eTG6u6hZUNMnp2UxbspehdlvIbnQfDl6mKSthjGGLq4FpQohlQEcgWUp5xQjHLZF9uzGwfRbDHLYyZUl9fp3WDQ9nNQLTmN5Zc5SoyzeMeszWdavw9qA2ZW43YsQI3n33XR544AEOHTrEhAkT2LZtGwAtW7Zk69atODg4sGHDBl577TVWrlzJvHnz8PLy4tChQxw5coTg4ODC46WmptKpUyc++OADXnrpJRYtWnTLbNBHHnmEuXPnMmvWLEJDQ0lPT6ekpRnfeecdunXrxltvvcXatWtZuHAhAMeOHWP58uXs2LEDR0dHpk6dyvfff1/4CUCpvPLyJC/+dJDGybup6ZgAHcaZ7FxlZkEhxFKgB+AjhIgB3gYcAaSUC4B1wADgNJAGjDdVsIWqN4JG3Rl/fQez4gbx0oqDfD6qnborbyMCAwM5f/48S5cuZcCAAbc8l5yczOOPP86pU6cQQpCdnQ3A9u3bee655wDw9/cnMDCwcB8nJ6fCPviQkBD++uuvUs9/8eLFwslFt9u6dSurVq0CYODAgXh5eQGwceNGIiIiaN++PQDp6enUrFmzvJeu2KD5W87wV9Q1NjeIgLQa0Ly0QYN3p8yELqUcWcbzEnjaaBEZqtsLuCRH80pyMz744wxfbT/Hk/c0NnsYtsqQlrQpDR48mBdffJHNmzcTHx9f+Pibb75Jz549+fnnnzl//nxhV0hpi507OjoWvtnb29uXmKwL1K9fHxcXF/Ly8gDuGPFSXMNBSsnjjz/ORx99ZND1KZXD9lNxfPLnCR4IrINv97cgORrsTTfz1noXuGjSE9o9xpM9WtCvTW0++v04u8/Gl72fYhUmTJjAW2+9RUBAwC2PJycnF96wDAsLK3y8W7du/PjjjwBERUVx+PDhcp3P09OTmze1+zHu7u40atSIiIgIAFauXFm43b333sv3338PwO+//05iYiIAvXr1YsWKFVy/fh2AhIQEg0fKKLbpUlI6zy47QJMaHvx7aCCiXltoPdik57TehA6QnoTYu5CPH6iPr7cb037Yz9VkNX7YFtSvX7+wC6Wol156iVdffZWuXbvectNx6tSpxMbGEhgYyL///W8CAwOpWrWqwecbN24cU6ZMITg4mPj4eF5++WWee+457rnnHuzt7Qu3e/vtt9m6dSvt2rXjzz//pGHDhgC0bt2a999/n759+xIYGEifPn24csWkt5IUC5aZk8vUJRFk5eSxYEw73Le9D1fL18ioCFHaR1VTCg0NlXe9wMW1ozC/C9z/Eacaj2XI5ztoUduTZZM64exgX/b+yi2OHTtGq1at9A6jQnJzc8nOzsbFxYUzZ87Qq1cvTp48iZOTU7mPpWq5/MOa/yb09MrKQyzbF82CMSH08zgDYQNgyDxoO/qujy2EiJBShhb3nHW30Gu1gXqhEBFGs5oezBoWxIGLSby7JkrvyBQzS0tLo1u3bgQFBfHQQw8xf/78CiVz0FYqUqsVKRW1dO9Flu2L5umeTejnXxsiwsC5KrR5yOTntv6xfiHjYPU0iN7DgIBOTO7emC+2nCWofjUeba/+U1YWnp6eRlvS0M3NzSjHUSqfAxcTefvXo9zbvAb/16cFpCVA1K/Q7jFwMv3flXW30AH8HwYnT+1dEJjetwXdmvrwxi9HOBidpGtoinVKTU0lNTVV7zAUKxN7M5OnluynVlVn5owIxt5OwKHlkJsJIY+bJQbrT+hO7hA4DFLjQEoc7O2YM7ItNTydmbIkgtibmXpHqFiZmJgYYmJi9A5DsSJZOXk8/f1+ktKzWDAmhGpu+d19eTnQpBfUDij9AEZi/QkdoP/HMGYF5I8Pru7uxBdjQ0hMy+LpH/aTnZunc4CKNWnYsGHh6BVFMcQHa6PYez6Bfw8NpE3dIqOrujwDY1eZLQ7bSOj2+bcC8lvpAP71qvLvoYHsPZfAB2uP6RicYm1cXV1xdXXVOwzFSvwUHs03uy4w8Z5GDAkuUtQt7lRhPjIX20joAKc3wictIOafG2NDguvxRLdGhO08z0/h0aXsrFiKDz74gDZt2hAYGEhwcDB79uwxewwpKSmkpKTc8lhYWBjTpk27Y9uwsDCEEGzcuLHwsZ9//hkhBCtWrDB6bAMGDCApKYmkpCTmzZtX7v03b95sklLEldXB6CRe/+UIXZt683K/lv88kZ4EC+6Bje+aNR7bSegNOoCDC0QsvuXhV/u3pEsTb17/5QiR6iapRdu1axe//fYb+/fv59ChQ2zYsMHkwweLq4h46dIlLl26ZPAxAgICWLp0aeHPy5YtIygoyCjx3W7dunVUq1atwgldMZ7rNzOY/F0ENT2d+d/IdjjYF0mnh36EnHSTzwy9ne0kdGdPCBgGR1ZCemLhww72dnw+qh21qjgz+btwrt9QM0kt1ZUrV/Dx8cHZ2RkAHx+fwrU9169fT8uWLenWrRvPPvtsYSuzoARuAX9/f86fPw/Agw8+SEhICG3atCmsighaOd233nqLjh07smvXrjvK3tavXx9fX18WL15M8+bN6d69Ozt27Cgx7nvuuYe9e/eSnZ1NSkoKp0+fvqXa47vvvkv79u3x9/dn0qRJhXVn9u3bR2BgIJ07d2b69On4+/sDWqv/4Ycfpl+/fjRr1oyXXnqp8FgFC2q88sornDlzhuDgYKZPn35Hy3vatGmFpRGKvnYFhcVAG80zYcIE2rdvT9u2bfn1118N/l1Vdpk5uTy1ZD/J6dksHBtKdfcicx6khPCvoW47qNvWrHHZTkIHCB0PORlwcNktD3u5O7FwbCg30nOYsiSCzBxVp9ogiwfe+bV3kfZcVlrxzx/Q6pyQGn/nc2Xo27cv0dHRNG/enKlTp7JlyxZAK441ceJE1qxZw7Zt27h69apB4X/99ddEREQQHh7OnDlzCot8paam4u/vz549e/D29i4sexsZGYm9vT0rV64kMTGRt99+mx07dvDXX38RFVXyZDUhBL179+aPP/7g119/ZfDgW1tl06ZNY9++fYULd/z2228AjB8/ngULFrBr165bygsAREZGsnz5cg4fPszy5cuJjr61y3DmzJk0adKEyMhIPv744xJjK+21++CDD7jvvvvYt28fmzZtYvr06Wq4pgGklMxYfZSIC4nMGhZE67pVbt3g4m6IPQahE8wem20l9DpBUC8EIr6542ZEqzpV+OTRIPZfTOLNX46UWp1P0YeHhwcREREsXLiQGjVqMHz4cMLCwjh+/DiNGjWiWbNmCCEYM2aMQcebM2dO4aIW0dHRnDp1CtAqLg4dOhS4textcHAwGzdu5NixY2zevJkePXpQo0YNnJycGD58eKnnGjFiBMuWLWPZsmW3LMgBsGnTJjp27EhAQAB///03R48eJSkpiZs3b9KlSxcARo0adcs+vXr1omrVqri4uNC6desKF/oq7bX7888/mTlzJsHBwfTo0YOMjAwuXrxYofNUJkt2X2DpXm0m6MDAYpZPPrQcnKtoc2TMzPpnit5u4Cfg5l04hLGoAQF1ePa+psz5+zSt6lRhfNdGOgRoRcavLfk5J7fSn3f3Lv35Etjb29OjRw969OhBQEAA33zzDcHBwSXWundwcCgscwv/lLrdvHkzGzZsYNeuXbi5uRUmLAAXF5fCFnFxZW9PnDjBhg0bylVfv0OHDhw5cgRXV1eaN29+SzxTp04lPDycBg0aMGPGDDIyMspsUBR0OxW8JmWV/C3pdYDiy/2Cdu0rV65UNWvKYefpOGasiaJ3q5r8q08Jr9uAj7XWuZO7eYPD1lrooPVZVSt5DPHzvZvTt3Ut3l97jO2n1OK+luTEiROFrWjQuh18fX1p2bIl586d48yZMwC33ID08/Nj//79AOzfv59z584BWpldLy8v3NzcOH78OLt37y72nMWVvRVCMHDgwMJa7NnZ2fz0009lxv/RRx/x4Ycf3vJYQWL18fEhJSWlcOSLl5cXnp6ehXEtW3ZrN2FZipb7BfD19SUqKorMzEySk5MLR92U9trdf//9/O9//yt8czlw4EC5YqhsLsanMfWH/TT2cefT4cHY2ZXwhm/vCHUCi3/OxGwvoQNcPQI/jNDGpd/Gzk7w3+HBNK3hwdM/7Od8nOoztBQpKSk8/vjjtG7dmsDAQKKiopgxYwYuLi4sXLiQgQMH0q1bN3x9fQv3GTp0KAkJCQQHBzN//vzC1nG/fv3IyckhMDCQN998k06dOhV7zuLK3iYkJODn58eMGTPo3LkzvXv3pl27dmXG379/f3r27HnLY9WqVWPixIkEBATw4IMPFq5oBPDVV18xadIkOnfujJSyXOV+vb296dq1K/7+/kyfPp0GDRoUrqk6evRo2rbVbsaV9tq9+eabZGdnExgYiL+/P2+++abB569sUjJzmPhtOFLCosdC8XQpZpGKvDz4ZtAd9/DMSkqpy1dISIg0mWvHpHy7ipTbPi1xk4vxqTL4nT/kfbM2yeT0LNPFYkWioqL0DsEgmzZtkgMHDjTZ8ZOTk2VycrLJjl/g5s2bhd9/9NFH8tlnnzX5OcvLWv4mTCknN08+EbZXNn51rdx2MrbkDU//reWdyGUmjQcIlyXkVdtsoddsCX73aEOH8oof0dKguhvzx4RwIT6NZ344QI4qD6Dku3LlilkWp1i7di3BwcH4+/uzbdu2WxauVizHrD9PsOHYdd4e1JpuzXxK3nDfl9r9uzYPmi2229lmQgdo/wQkXdBmkJagU2Nv3nvQny0nY/no9+NmDE65Gz169Cgc+mcKjRo1olEj098wHz58OJGRkRw5coS1a9dSo0YNk59TKZ+fD8Qwf/MZRndsyNhOviVvmHwJTqyDtmPBwbnk7UzMdhN6ywfAo5b2rlmKkR0aMq6LH19tP8eyvWrIlgJOTk4VXhxDsR37Lyby8srDdGpcnRmD25Q+6ikiTBsqHTrebPEVx/aGLRawd4R7/gXZadoLXcov442BrTgTm8IbvxzB19udzk28zRioYmmSk5MBynWTUrEtMYlpTPo2nDpVXZg/OgRH+zLavo3uBXsn8PIzS3wlsd0WOkDHydDthVKTOWjlAeaOaoefjztPfR+hRr5UclevXjV4Nqpie1Iyc3jym3Ayc/L46vH2eLkb8Gmt0T3QfbrpgyuDbSd0gJwsrb5Lduk1XKq6OvLV46EIYMI3+0hOyzZPfIrFady4MY0bN9Y7DEUHuXmSZ5ce4NT1FOaPDqFpTY+yd9r3JSSeN3lshrD9hH5xJ6yYAFG/lLmpr7c7X4wNJTohjak/RKiFMXRgb29fOPJj2LBhpKWllWv/6dOn06ZNG6ZPL39rqWBSkKOjI46Ot44z9vPzIyAggODgYAICAgwqZFVQSEuxHh+uO8bfx68zY3Cb0ke0FLh+DNb+C47+bPrgDGD7Cb1Rd/BuBnsXlr0t0KFRdT56OJAdp+NVzRcduLq6Fo78cHJyYsGCBQbtVzA1/osvvmD//v2lFqwqSUFCL6g3frtNmzYRGRnJihUrePbZZ8t9fMWyfbf7Al9tP8f4rn6lj2gpau9CsHeGto+ZNjgD2X5CFwI6TIJLERATYdAuj4TUZ1rPpizbF83CrWdNHKBSknvuuYfTp0+XWOY1LCyMYcOGMWjQIPr27cvgwYNJTU2lY8eOLF++nNjYWIYOHUr79u1p3759YQnclJQUxo8fT0BAAIGBgaxcuZJXXnmF9PR0goODGTt2LNeuXSsxrhs3buDl5VX4c0lleosqrZTv66+/XlhErOC8165d46GHHiIoKIigoCB27twJcEep3+LquSvlt+VkLDNWH+W+ljV5Y2Brw3ZKT9JmhQY8otUusgQlzTgy9ZdJZ4reLj1Zyg/qSrlqssG75Obmyae/j5B+r/wmfz982YTBWY7bZwV2795dLl68WEopZVZWluzevbv87rvvpJRSpqamyu7du8tly7RZcUlJSbJ79+5y5cqVUkopY2NjZffu3eXq1aullFJeuXLFoBjc3d2llFJmZ2fLwYMHy3nz5slXX3218LyJiYmyWbNmMiUlRS5evFjWq1dPxsfH37G/lFKOHDlSbtu2TUop5YULF2TLli2llFK+9NJL8rnnnivcLiEh4ZZ9s7KyZFbWrbOHfX19pb+/v2zTpo10dXWVa9asKXyu4PxpaWmyTZs2Mi4urnCf2NjYUrcBCl+j6dOny/fee09KKeWjjz4qP/30UymllDk5OTIpKUlGRUXJBx54oDC2p556Sn7zzTcGva4VVRlmih6/ckO2eWu97PfZVnkzI9vwHXfN02aGXtpvuuCKQSkzRW132GJRLlUgaCTE7NVmjtrZl7mLnZ1g1rAgLiWl8/zySJZWcaFtQ68y91PuTkErGbQW+hNPPEGXLl1YvXp14UIWRcu89unTh+rVqxd7rA0bNtxSx/zGjRvcvHmTDRs23FIMq2hrG7ij/7zApk2b8PHx4cyZM/Tq1YsePXrg4eHBnDlz+PlnrQ+1oEyvt/etLbaStnFycipcmCIkJIS//voLgL///ptvv/0W0O4rVK1ale+++66w1G/Ba1WzZs3SXk6lDNdvZDB+8V7cnOz5elwoHs7lSIkp18G3m9kXsShN5UjoAH3eBUfXMocwFuXiaM+ix0J5eN5OnvwmnF+e7kqD6m4mDNKybN68ufB7R0fHW352c3O75eeqVave8rOPj88tP9euXdugcxb0oRclSyjzumfPHtzdSy5RmpeXx65du+5Y8FlKWeokkcREbcWr2xN9gSZNmlCrVi2ioqJIS0srsUxvgdJK+To6OhbGUlaZXFlMqV+l4lIzc5jwzT6S0rP5cXJn6lQt58Lgvd8usbSIXmy/D72Ak5uWzDNTILf02tJF+Xg4s3h8e3LyJOMW71XDGXVQ0TKvffv2Ze7cuYU/F7xR3P54QQJ3dHQkOzub69evF5bTLc7169c5d+4cvr6+BpXpNbSUb1G9evVi/vz5gLbu6Y0bN4ot9VvRhS8qu9w8yXPLDhB1+Qafj2qHf71yTiJLzH/dDfi0b04GJXQhRD8hxAkhxGkhxCvFPO8lhPhZCHFICLFXCOFv/FCN4Pox+G8rOFG+hRea1PDgi7EhXExIY9J34WoJOzOraJnXOXPmEB4eTmBgIK1bty4cMfPGG2+QmJiIv78/QUFBbNq0CYBJkyYRGBjIjBkzaNKkyR3H69mzJ8HBwfTs2ZOZM2dSq1Ytg8r0GlrKt6jZs2ezadMmAgICCAkJ4ejRo8WW+jVHETFbI6Xk3TVH2XDsOu8MbkPPluXstko4C7ODtOn+FkYUtHpK3EAIe+Ak0AeIAfYBI6WUUUW2+RhIkVK+I4RoCXwupexV2nFDQ0NleHj43cZfPnm5MCcYqjaA8evKvfuvkZd4blkkg4Pq8llpBe6t1LFjx2jVqpXeYSgWxBb/JhZuPcOH644z6d7GvDagAte2/lVtuOLzR6BKMUvQmZgQIkJKGVrcc4a00DsAp6WUZ6WUWcAyYMht27QGNgJIKY8DfkKIWncRs2nY2WtDGC/sgCuHyr37kOB6vNSvBasPXuY/f5wwQYCKJUhISCAhIUHvMBQTWHPwMh+uO84DgXV4pV/L8h8g8yYcWAKtH9QlmZfFkIReDyi65HhM/mNFHQQeBhBCdAB8gfq3H0gIMUkIES6ECI+Nja1YxHer7RhwdIM9X1Ro96e6N2FMp4Ys2HKG73adN25sikWIjY1Ft79PxWT2nI3nXz8epEOj6swaFlSxT9iRP0DmDej0lPEDNAJDEnpxV317P81MwEsIEQk8AxwA7rjzKKVcKKUMlVKG6lb72dVLG8J4+CdITyz37kIIZgxqQ+9WNXl79VH+OGpbRZzK6oKrDJo2bUrTpk31DkN3tvS3cPLaTSZ+G06D6q4sHBuCi2MFb2ZG/gD1QqF+sT0eujMkoccADYr8XB+4XHQDKeUNKeV4KWUw8BhQAzhnrCCNrtvz8MQfWnKvAAd7O+aMbEtg/Wo8u/QA4edt4+O5i4sL8fHxNvUfuSLs7e2xt7es0QvmJqUkPj4eFxcXvUO5a5eT0nn86724ONrzzYQOVHO7i1r3j6+BB+cbLzgjM+SmqAPaTdFewCW0m6KjpJRHi2xTDUiTUmYJISYC90gpSy1uoMtNUSOLT8nkkQW7SEjNYuVTnWla01PvkO5KdnY2MTExd4yjrmxSUlIAbVp+Zebi4kL9+vVLnGhlDZLTsxm2YCeXkzL4cXJnWtetUvGDlbGugrmUdlO0zIlFUsocIcQ04A/AHvhaSnlUCDEl//kFQCvgWyFELhAFPGG06E0l4wasfwWa9YE2D1XoEN4eznwzvgMPz9/JY1/tZeXULuWfnGBBHB0dzbL0mqXr0aMHcOvEKsX6ZGTnMvGbcM7FpRI2vsPdJfPYE7DiCXjwc6gTZLwgjazMFrqp6N5Cz8uDzzuAswdM3HRX77xHLiUzYuFu6lZz4afJXajqZr0tGkX7pAIllwBQLF9Obh5Tv9/PX8euMWdEWwYF1b27A655DiKXwv9FgbsBZXVN6G6HLdomOzvoNAUuH4CLZc/cK41/vaosHBvC+bg0nvhmH+lZauKRNSuuHrpiPaSUvPHLEf6MusbbD7S++2SeGq9VVQwarnsyL0vlTeigjXZxqQa7P7/rQ3Vp6sOnw4OJuJjItB/2q8UxrFhYWBhhYWF6h6FU0Cd/nmTZvmie7tmEcV2N0IUY8TXkZECnqXd/LBOr3AndyR1CJ8DxtUZZQmpgYB3eHeLPxuPXeXnFIfLyKvdoEWulErr1+nLbWeZuOs3w0Aa82LdF2TuUJScT9i6CJr2gpuXPmK081RZL0mEiZKeDg3GGZ43t5Etiahb//esk1dycePOBVqVW9lMsj7oZap1WRsTw/tpj9GtTmw8e8jfO/zthp1Vq9bKOwQIqoVepC/1nGvWQz9zXlMS0LL7ecQ4vN0ee6dXMqMdXFOVWf0Vd46WVh+ja1JvZI4NxsDdS54O9IwSNMM6xzEAl9AJnt0BWCrQceNeHEkLw5sDWJKdl88lfJ6ni6sjjXfzuPkbFLBYtWgTAxIkTdY5EMcTO03E8/cN+/OtW4YuxoTg7GGlS2MU9EL0b2k/Uym9bAZXQC2z9GBLOQbO+2rvyXbKzE/znkUBuZubw9uqjeDg7MDTkjvI2igVavnw5oBK6NThwMZEnvw3Hz9uNsPEdyrfiUFm2faKtRdxhkvGOaWKV+6ZoUV2egRsxcPQXox3Swd6O/41sS9em3kxfcZD1R1TtamuwYcMGNmzYoHcYShmOX73BuMX78PFwZskTHfFyv4sp/be7fhxO/QEdJ2srnVkJldALNO0DPi1g5xxtiq+RuDjas3BsKEENqvHM0gNsPlHySjiKohjmTGwKY77cg4ujHd8/2ZGaVYxcc2bXXHBwhVDLn/RelEroBezsoMs0uHoIzm016qHdnR0IG9eBZjU9mfxdBLvOxBv1+IpxzZs3j3nz5ukdhlKC6IQ0Ri/ag5Tw/ZOdjL/O781rcGg5BI8Cd++yt7cgKqEXFfAo1AqANOMn3Kpujnz3RAcaVnfjiW/2EXGh/KV7FfNYs2YNa9as0TsMpRhXktMZ9eVu0rNzWfJkR5rWNEEBtYxk8O0CnZ82/rFNrPLWcimJiSuqXb+RwaNf7CI+NYvvn+xIYP1qJjuXotiS6zcyGLFwN9dvZvL9kx0JalBN75B0oWq5lIcQkJsD0ftMcviaVVz4fmInqro6MvarvRy5lGyS8yiKLYlLyWTUl3u4eiODsPHtTZfML+yC5EumObYZqIRenC3/hsX9TfaLrVfNlaUTO+HuZM/Yr/Zw/OoNk5xHqZjZs2cze/ZsvcNQ8iWkZjHmyz3EJKbx9bj2hPpVN82JcrNh1ST4ebJpjm8GKqEXp+0YkHmw23Q3xhpUd+OHiZ1wcrBj9KI9nLh602TnUspn48aNbNy4Ue8wFCAxP5mfi0vlq8fb06mxCW9SHv0Fki9aZd95AZXQi+PlC/5DISKsQuuOGsrPx52lEzthbycYtWg3J6+ppG4JVq9ezerVq/UOo9JLTM1i9Jd7OB2bwsLHQuna1ISla6WEHbOhRktodr/pzmNiKqGXpOuzWimAfV+Z9DSNa3iwbJKW1EcuVEldUeDWZL7osVC6NzfxovKnN8K1w9DlWW0Is5Wy3shNrXYANO0NZzaZ/FS3J3XVp66vWbNmMWvWLL3DqLQSUrMYZc5kDloyr9YQAoaZ/lwmpIYtliY1Hly9zPaOfTY2hVGL9pCZo42xbVO3qlnOq9xq6NChAKxcuVLnSCqfuJRMRi/aw/n4VBY9Fsq95kjmBbLTrWKaf2nDFlVCN0TmTW0asL3pa5ldiE9l5MLdpGblsuSJjgTUV0ldqRyu38hg1Jd7uJSYzlePh9LFlH3mRSVe0O6bWQk1Dv1uxJ2GT/3h6M9mOZ2vtzvLJ3fG08WBUV/uVjNKlUrhclI6wxfu5nJSOmHj25svmV8/DrODtAWgbYBK6GWp3hg868D2/0KeedYJbVDdjeWTO+Pt7sTYr/ao2i9mNnPmTGbONO6iJ0rJLsSnMmzBLuJuZvLdEx3oaMqhibfb/qnWzdKsr/nOaUIqoZfFzg66vQDXo7RymmZSr5orP07uTL1qroxbvFdVaTSjyMhIIiMj9Q6jUjh9PYVHv9hFalYOP0zsRIiviSYNFSfxAhz+CULGW10RrpKohG4I/6HaHfCts4xaWrcsNau4sHxyZ5rW9GDit+GsPaTqqZvDsmXLWLZsmd5h2Lwjl5IZ/sUucvNg+aTO5r9ftHOOtmaoFU8kup1K6Iawd4Cuz8GlcLh8wKynru7uxA8TOxFUvxrPLN3Psr0XzXp+RTGFPWfjGblwNy6O9vw4uRMtanuaN4DsDO2+WPBIqFrPvOc2IbUEnaGCx0DtQKjXzuynrurqyHdPdGTKkgheWXWYGxnZTLq3idnjqCzee+89AN58802dI7FNfx+/xlNL9lPfy5XvnuhI3Wo6DBV0dIFp4Vr9FhuiErqhHF2gQQftexOX2C2Oq5M9ix4L5YUfI/lw3XHiU7J4pX9LhJnjqAxOnDihdwg2a9X+GKavOESrOp58M74D3h7O5g8iJ0tbN9jNjP31ZqISenlteAfiTsKI781+aicHO+aMaIuXmyNfbD1LXEoWM4cG4Gives6MacmSJXqHYJMWbT3LB+uO0aWJN1+MDcHT5e4XY6+Qrf+B0xtg/HqtoWZDVEIvL2cPOP6b1pdet63ZT29vJ3hviD8+Hs58tuEUiWlZzB3VFjcn9atULFNenuTf64/zxdazDAiozafDg3F2sNcnmPQk2PMFNO5hc8kc1E3R8ms/EVyqaiNedCKE4PnezXn/QX82n7jOyEV7iE/J1C0eW/PWW2/x1ltv6R2GTcjMyeWFHyP5YutZxnRqyP9GttMvmQPsXQiZN+De6frFYEIGJXQhRD8hxAkhxGkhxCvFPF9VCLFGCHFQCHFUCDHe+KFaCJcq0PEprZV+9YiuoYzp5MuCMSEcv3KDofN3ciE+Vdd4bEV0dDTR0dF6h2H1bmRkM+7rffwaeZnp97fgvSH+2NvpeM8n86a2xkHz/lAnUL84TKjMhC6EsAc+B/oDrYGRQojWt232NBAlpQwCegCfCCGcjByr5eg0BZw8YdsnekdC3za1+WFiJ5LTs3l43k72X1SlAu7W4sWLWbx4sd5hWLXLSek8umAX+84n8N9Hg3i6Z1P9b+Af+F5b38BGW+dgWAu9A3BaSnlWSpkFLAOG3LaNBDyF9hvzABKAHKNGaklcveCh+dDLMj6Wh/h6sfKpLrg7OzBy4W7WHVYTkBT9HI5J5sHPd3ApMZ2w8R14uF19vUPShI6HkcugfojekZiMIQm9HlD082dM/mNFzQVaAZeBw8BzUso7Cp8IISYJIcKFEOGxsbEVDNlCtBoE1RvpHUWhxjU8+HlqF9rUrcLU7/fzxZYz6FVJ09q9+uqrvPrqq3qHYZU2RF3j0S924Whvx4qnutCtmZmKbJVFSnBwhhb99Y7EpAxJ6MV9Tro9U9wPRAJ1gWBgrhCiyh07SblQShkqpQytUcOMdY5NJfEC/DAcrkXpHQkA3h7O/DCxEwMD6/DR78d5eeUhsnLMU1DMlsTHxxMfrwqilYeUkkVbzzLxu3Ca1fLg56e7mH/2Z0kyU+CLe+Gk+Wox6cWQsW4xQIMiP9dHa4kXNR6YKbUm4WkhxDmgJbDXKFFaKmdPOL9dG9c6LEzvaABwcbTnfyPa0sTHnTl/n+Z8fBoLxoRQ3d12b2kY28KFC/UOwapk5eTx+s+H+SkihgEBtZk1LMiyhtHu+xKuHgJX25tIdDtDWuj7gGZCiEb5NzpHALevoHsR6AUghKgFtADOGjNQi+RWHTpO1lYLv35M72gK2dkJ/q9vC2aPCCYyOokhn2/nxFW1VqlifHEpmYz5cg8/RcTw7H1NmTuynWUl88wUrQhX097QoL3e0ZhcmQldSpkDTAP+AI4BP0opjwohpgghpuRv9h7QRQhxGNgIvCyljDNV0Bal8zRwcoct/9Y7kjsMCa7H8kmdyMjO46F5O1h/RN0sNcSLL77Iiy++qHcYFu/IpWQG/287B2OSmD0imP/r2wI7PYclFmffIkiLh+53jLa2SQaNQ5dSrpNSNpdSNpFSfpD/2AIp5YL87y9LKftKKQOklP5Sysozd9qtOnR6SqvcpvO49OK0bejFb890o3ktT6Ys2c9//zxBXp66WVqa9PR00tPT9Q7Dov0aeYmh83cCsGJKF4YEW2DFwsybsGM2NO1TKVrnoNYUNY6C6cQdJ4NrNb2jKVZGdi5v/nKEnyJi6NmiBp8Nb0tVN51qaShWKzs3j4/WHefrHefo4FedeWPa4aNHgS1DSAmn/oQq9aC2v97RGI1aJFoBtJEIS3Zf4N3foqhT1ZX5Y9rRpq5ahFoxzPWbGUz7/gB7zycwrosfrw1ohZODqh5ibmqRaHM59Rf8brl9dUIIxnb2Y/nkzmTl5PHwvJ38GK6muN/u+eef5/nnn9c7DIuy+2w8D8zZzuFLycweEcyMwW0sO5lvnQUb3zXrCmOWwIJ/I1bo2lHYMx8u7tY7klK1a+jFb892I8TXi5dWHOJfPx4kLct2J/YqFZeXJ5n79ylGLdqNh7MDPz9tof3lRaXGwbb/QsI5s69boDfV5WJMWakwOwhqtITH11j8H1NunmTOxlPM+fsUTWt48PnodjSvZSGTQRTdxaVk8n8/HmTryVgGB9Xlw4cD8HC2oCGJJfnjda0I19Q9UKO53tEYnepyMRcnd7jnRTi/Dc78rXc0ZbK3E7zQpznfTehIYloWg/63nSW7L6iSAQrbTsXS77Nt7D4bz/sP+jN7RLB1JPPkS7B3EQSOsMlkXhaV0I0tdDxUbWhV/Xfdmvnw+3P30rGxN2/8coQpSyJISsvSOyzdPP300zz9tO2sBF8eWTl5fPT7McZ+tRcvN0d+fborYzr56l8p0VBb/wMyD3pY7r0sU7KCt1wr4+AM93+glemUeSB0LOZfDjU8nQkb156vtp/jP38c5/7PtjJrWBD3NLOBmjvl5Oqqw6LFFuDktZs8vyySqCs3GN2xIW8MbI2rk3X8/RZqPxHqtgMvX70j0YXqQ1fucORSMi8sj+TU9RTGdfHj5X4tre8/tmKwvDxJ2M7zzFx/HE9nB2YODaRP61p6h6WUQPWh6yE3B3YvgCOr9I6k3PzrVWXNM92Y0LURYTvPM3DONiIuJOgdlmICF+PTGPXlbt79LYp7mvqw/vl7rTOZXzkIK56Am9f0jkRXKqGbip09HFoOf74J2Rl6R1NuLo72vDWoNT882ZHMnDweWbCL93+LIiM7V+/QTG7SpElMmjRJ7zBMKi9P8u2u8/SbvZWjl27wn6GBfPl4KDU8LXTWZ1k2vAOnN2hdnpWYSuimIgT0eQduxMDeL/SOpsK6NPXhjxfuZVSHhny5/Rz9PtvKztO2XXfN29sbb29vvcMwmdPXUxi+cBdv/XqUUL/q/PHCvTzavoH13Pi83dnNcGYj3PuixZbeMBfVh25qS4ZCzD547qC2dJ0V23k6jld/PsyF+DSGhdTntQGt8FJ11q1GZk4uCzaf5fNNp3F1suf1ga0YFlLfehM5QF4eLOqpVVScFg6OLnpHZHKqD11Pvd+BjBvazDUr16WpD388fy9P9WjCqgOXuO+Tzfy4L1pVb7QC20/F0f+zbXy64ST3+9dmw/9159FQK26VFzi6Cq5EQs/XK0UyL4tqoZvD1o+hfnto3EPvSIzm+NUbvPHzEcIvJBLi68W7Q9rYTKGv8ePHA7B48WKdI7l7V5MzeH9tFL8duoKvtxvvDG5DjxY19Q7LeFKuQ0QY3PMv7b5VJVBaC12NQzeHe6frHYHRtaxdhR8nd2bl/hg++v04D/xvOyPaN+BffVtYbjlVAzVo0KDsjSxcRnYuC7eeZf7mM+RKyQu9mzO5e2NcHG0s6XnUhO4v6R2FxVAtdHPJSIbN/4ag4VAnSO9ojCo5LZvZG0/x7a7zuDra8/R9TRnXxc/2kocVyMuTrDl0mf+sP8GlpHT6+9fmtQGtaFDdTe/QjCs9CVZNhF5vQe0AvaMxK9WHbglkHhz8Af58w2pKAhiqqpsjbw1qzfrn76V9o+rM/P04983azMqIGHJV/7rZ7Dgdx+DPt/PcskiqujqybFIn5o8Jsb1kDrDtE61ctY39X7pbKqGbi6uXtq7hua3aKio2qGlND74e154fJnbEx9OZf/10kAGzt7H+yBWrKvg1ZswYxowZo3cYBtt/MZExX+5h9Jd7SEzN5tPhQfz2TDc6NbbRoZeJ52HPAggeBXUC9Y7Goqg+dHMKnQB7F2qTjZr0AnvbfPm7NPHhl6ldWXv4Cp9uOMmUJftpU7cKz/duTu9WNS1+ZEWLFi30DsEgB6OT+GzDSTadiMXb3Yk3BrZiTCdf2+/q2vCOViPpvjf0jsTiqD50czv2GywfDQM/gfZP6h2NyeXk5vFr5GVmbzzFxYQ0Wtb2ZGrPpgwMqIO9pa0QbwWklOw+m8Dnm06z/XQc1dwcmXRvYx7v7Ie7NZS3vVsxEfDlfXDvS3Df63pHowu1pqglkRI2fQCBw8Gnmd7RmE1Obh6rD15m3uYznL6eQsPqbkzo6sew0AaVIxHdpZzcPNYfvcqibec4GJ2Ej4czE+9pxOhOvtZRp9xYsjMg/Gto9xg4e+gdjS5UQlcsRl6e5M+oqyzcepb9F5Oo4uLAiA4NGd2xIb7e7nqHB8CIESMAWLZsmc6RQEJqFj+FR/PtrgtcSkrHz9uNJ7o1YlhoA9vvWlGKpcahW6LkS7D+Zeg1A3ya6h2N2djZCfr516Gffx0iLiTy9fZzfLX9HIu2neXeZjUY3bEhPVvWxNFev/v1wcHBup0btG6V8AuJLN1zkd8OXyErJ4+OjaozY3AberWsiV1l7KrKSoXvHoLuL0PTXnpHY7FUQteLvSOc2Qy5b8Ao/VuCegjx9SLE14uryRks23eRpXsvMum7CHw8nHgwuB5DQ+rTsran2W+ivvKKPqvdXEpK55cDl1gREcO5uFQ8nB0Y0b4BYzr5qrVed8yG6D3gaINDMI1IdbnoaftnsOFtGLNKtTqA7Nw8tp6M5afwGDYev0Z2rqRpTQ8GBdZlYGAdmta0vT7T6zcy+P3IVdYcvEz4hUQAOjaqzrDQBvT3r63uLwAkRcPcUGgxAIZZfzmGu6X60C1VTiZ83lGr4Txlu9ZqVwCt73jd4SusOXiZvecTkBIa13CnT+ta9G5Vi7YNquFgom6ZoUOHArBy5UqjH1tKyYlrN9l47Dp/Rl3jYHQSAC1rezIoqC6DAuvS0Fu1Qm+xYgIcX6tVU6xm/WUZ7pbqQ7dUDs5w/4ewbKR2577jZL0jshjV3Z0Y08mXMZ18uZqcwZ9RV/kr6hpfbTvHF1vO4unsQOcm3nRr5kOHRtVpXtPTaH3LnTt3NspxQEvgMYnp7D2XwI4zcWw7FUfszUwAgupX5cW+zbm/TW2aVfYulZJcPgBHVmp95yqZl0m10PUmJez7EgIesfp66eZwIyOb7ae0xLj1ZCyXktIBqOLiQIivF4H1qxFQryr+9apSq4qz2fvfk9KyOHr5BocvJXM4JpmIC4lcvaGtWOXl5ki3ZjW4p5kP3ZvXoFYVVe61TFLCsdXQtA84qU8uoLpcrIeU2kpHikGklEQnpLPvfAL7zicQcSGRM7EpFJSPqeLiQLNanjSt4UFDbzfqe7lS38uNmp7O1PB0rtCwv6ycPOJTM4m9mcmlxHSiE9OITkjnTGwKp66nFLa+Aep7udK2oRcd/LwI9atOi1rG+xRRKeRkgYNaQOV2qsvFGsSfgZ/GaTNIG3TQOxqrIISgobcbDb3dGBpSH4C0rByiLt/gyKVkTsemcOpaChuPXyMuJeuO/T2cHfB0ccDD2QF3ZwecHOxwtBdsnfsSEknXp/5Ddm4eaVm53MzI4WZGNjcycu44jqeLA41reNC9eQ2a1fSgVZ0qBNSrqlZzuhspsbCwu9Yl2eZBvaOxGgYldCFEP2A2YA98KaWcedvz04HRRY7ZCqghpVRLxRvKoxakxsG6F2HipkpTrN/Y3JwcCPWrTqhf9VseT8vKISYxnUuJ6cTezCQ2JZO4lExSMnJIydS+snLyyMjOo1rTtggBEnB1ssfHwxkPFwc8nR2o7u6Mj6cTPh7O1KvmSoPqblR1VTezjW7jDEi5BjVb6x2JVSmzy0UIYQ+cBPoAMcA+YKSUMqqE7QcBL0gp7yvtuKrLpRhHVmp39AfMgg4T9Y5GUfRxcQ983Re6PAt939M7Gotzt/XQOwCnpZRnpZRZwDJgSCnbjwSWlj9MhTYPa8vUbXwPbl7TOxpFMb/cHFj7f1ClvjayRSkXQxJ6PSC6yM8x+Y/dQQjhBvQDih3AK4SYJIQIF0KEx8bGljdW2ycEDPgEctJh9+d6R1Np9e/fn/79++sdRuV0bgtcOwL9Z1ba4lt3w5A+9OJuy5fUTzMI2FFS37mUciGwELQuF4MirGx8msJjq6F+sZ+oFDMYNGiQ3iFUXk17wdTdUKOl3pFYJUMSegxQdER/feByCduOQHW33D3f/Ikt6Ung6KpNQFLMZurUqXqHUDnFnwHvJlCzld6RWC1Dulz2Ac2EEI2EEE5oSXv17RsJIaoC3YFfjRtiJZUar5UF2P6Z3pEoiumdWA//C4HTG/WOxKqVmdCllDnANOAP4Bjwo5TyqBBiihBiSpFNHwL+lFKmmibUSsbdG/y6wrZZEHdK72gqld69e9O7d2+9w6g8MlNg7b+0bha/e/SOxqoZNA5dSrkOWHfbYwtu+zkMCDNWYArQbyac3gBrnoPHfwM7taa3OQwfPlzvECqXTR/AjUvwxJ9qZuhdUjNFLZlHTejzHqx5FiKXaMtuKSY3caKaA2A2l/bDngXaAupqhvRdU00+S9d2LPh2hXNb9Y5EUYwv9gRUawi939Y7EpugWuiWzs4ORi4F5yp6R1Jp9OjRA4DNmzfrGkelEDwS/IeqrhYjUQndGrhU1f5NvADJ0eDXTd94bNy4ceP0DsH2xZ6A68e0wlsqmRuNSujW5JentBEvT+8Bt+plb69UiEroJpaXC79Og/hTWqkL12p6R2QzVB+6Nen/b0hPgPWv6h2JTcvOziY7O1vvMGzX3oUQs1cbxaWSuVGphG5NagdAtxfg0DI48bve0disPn360KdPH73DsE3xZ2Dju9oKRIFqeKixqS4Xa3PvdDi+Thub3qCj6noxgSeffFLvEGxTXh78MhXsHGHQbLU6lwmohG5tHJzhoQUQsRjs1cIKpjBmzBi9Q7BNQmh1/oUdVC22YKtyl1RCt0Z1AuGBT/WOwmalpaUB4OamFiU2moL1cgMe0TsSm6b60K3Z1cMQ9oC2/qJiNAMGDGDAgAF6h2E7crO1v9OIML0jsXmqhW7N7Bwgei+sfkabfKT6JI3iqaee0jsE27J1FlzYDu2f0DsSm6cSujWr2Qp6z4A/XoX930DIOL0jsgmqOJcRxYTD1o+1ES3+D+sdjc1TXS7WruMUaNRdG5sef0bvaGxCcnIyycnJeodh/TJTYNVEqFIXBnysdzSVgkro1s7ODh6cr4142TlH72hswpAhQxgypLR10BWDnNmolat4aME/5SsUk1JdLragaj0Ytw5qtNA7Epvw7LPP6h2CbWg9BJ7dD15+ekdSaaiEbitq+2v/piVAcow2tFGpkIcfVn29dyU5RmuZ+3VVydzMVJeLrflpHPwwXEvsSoXExcURFxendxjWKS8XVk6EpSMhQ92HMDeV0G1N3/cgLU6rZiel3tFYpUceeYRHHlETYCpk6yy4uFO7Car6zc1OdbnYmjpB0PsdbSjjni+g05Sy91Fu8a9//UvvEKzTuW2wZaY2RDFIDf3Ug0rotqjTU3B+G/z5BtRvD/VD9I7IqgwaNEjvEKxPWgKsfAKqN4GB/9U7mkpLJXRbJAQ8OA/+eF1br1Epl6tXrwJQu3ZtnSOxIi7VoMsz0OQ+cPbQO5pKS0id+llDQ0NleHi4LueudHJzwM5elQYwkFpTtJwyU1QSNyMhRISUMrS451QL3dZl3oTvH4XmfbXFMZQyvfLKK3qHYD1Ob4BVk2DMSqjbVu9oKj2V0G2dkwd41tJWianbVlvDUSlVv3799A7BOiSehxVPQJV64NNc72gU1LBF2ycEDJ6r/YdbMQGSovWOyOJFR0cTHa1ep1Jlp8PyMYCEEUvAyV3viBRUQq8cnD1g+PdaXeofx0J2ht4RWbSxY8cyduxYvcOwXFLCb/+n1eN/eBFUb6x3REo+1eVSWfg01Yok/fE6pFwDL1+9I7JYb7zxht4hWLa8HMjLhu6vQPP79Y5GKUKNcqlscjK1dUkVpSIKlpKTUvuyUx/yza20US7qt1HZODhDThaseV4boaDc4ezZs5w9e1bvMCxP/Bn4up/2rxAqmVsgg34jQoh+QogTQojTQohix3QJIXoIISKFEEeFEFuMG6ZiVHnZELMPfpoAcaf0jsbiTJgwgQkTJugdhmXJSNYKbsWdAKESuaUqsw9dCGEPfA70AWKAfUKI1VLKqCLbVAPmAf2klBeFEDVNFK9iDE7uMOIHWHQffD8MntwI7t56R2Ux3nnnHb1DsCy52fDj45BwBsb+DNUb6R2RUgJD3mo7AKellGellFnAMuD25VxGAauklBcBpJTXjRumYnRevtrC0jcuw7JRauRLEd27d6d79+56h2EZpIR10+HsJnjgM2h0r94RKaUwJKHXA4oOyo3Jf6yo5oCXEGKzECJCCPFYcQcSQkwSQoQLIcJjY2MrFrFiPA06aCNfrkdB7DG9o7EYJ06c4MSJE3qHYRmyUuFKpDbLuJ0aymnpDBm2WFwBkNuHxjgAIUAvwBXYJYTYLaU8ectOUi4EFoI2yqX84SpG5/+wtsi06nIpNHnyZEDVcgG0OQzjfwd7NTLKGhiS0GOABkV+rg9cLmabOCllKpAqhNgKBAEnUSxfQTLfvQDsHaD9k/rGo7MPP/xQ7xD0d3477JoHDy9UhbesiCEJfR/QTAjRCLgEjEDrMy/qV2CuEMIBcAI6Ap8aM1DFxPLy4OxmOLke3GtC68F6R6SbLl266B2Cvq4dhaWjwLM25GbpHY1SDmX2oUspc4BpwB/AMeBHKeVRIcQUIcSU/G2OAeuBQ8Be4Esp5RHTha0YnZ0dPPI11A+FlU/C+R16R6SbI0eOcORIJf3zTYqGJY+Ak5tWQdGtut4RKeWgZooqt0qNh6/vh5tXYdyaSlkStdLWQ0+5rk0cSo2D8eugtr/eESnFUPXQFcO5e8Njv8LifnD5QKVM6B9//LHeIegjLR5kLoz+SSVzK6Va6ErxslL/KYmal6uteKTYppxMsHfSpvPnZoO9o94RKaVQtVyU8itI5ud3wIJukHRR33jMKDIyksjISL3DMI+sVPjuIW0BFFDJ3MqphK6UzskNki/BN4MgOUbvaMzi+eef5/nnn9c7DNPLSoMfhsPFXVCrjd7RKEag+tCV0tVtC4/9DN8+CGEPaDfLqtTVOyqT+uyzz/QOwfSy02HZSLiwAx76AgIe0TsixQhUC10pW70QGLNKG/2wuL82GsKGBQcHExwcrHcYpiOlVr/n7BYYMg8CH9U7IsVIVAtdMUyD9trol8gl4GbbZQL27dsHQPv27XWOxESEgLZjIGikSuY2Ro1yUSom+RJk3oCarfSOxOhsdhx6WoI2FLVpL70jUe6CGoeuGN/Pk7VFgkf9CA076h2NUc2dO1fvEIwvKRqWPAw3rsDzh9QMUBul+tCVihn8Py0pfDsETv6hdzRG5e/vj7+/DU2suX4MvuoLN6/BqOUqmdswldCViqneCCb8CTVaaEuTHViid0RGs3PnTnbu3Kl3GMZxYZc2nV/maiOU/LrqHZFiQiqhKxXnUQPG/aatYrNngTbL0Aa89tprvPbaa3qHYRxnNoK7Dzzxp5rOXwmom6LK3cvN1hYRdvfRZh4Ke3B00TuqCitYrahFixY6R1JBUmpLC1atp5VFzroJLlX1jkoxEjX1XzEte0ctmUsJKydC2EDt5puVatGihfUm86w0WDURFnaHlFitLLJK5pWGSuiK8QgBwSO1m3ALu0P0Xr0jqpAtW7awZcsWvcMov6SLWunjwyug01Pam6xSqaguF8X4rkVp08qTL8GA/0DIeC3ZWwmrHId+dgusGA+5OTD0S2jeV++IFBNR49AV86rVGiZu0lY+2vQhtH7QqobKff3113qHUH77vgQ3HxjxA/g01TsaRSeqha6YTl4eJJ2H6o21muoJ51SyMaaU65CTAdUaajelhb1a0LkSUDdFFX3Y2WnJHGDnHJjfRVtJPi9P37jKsGHDBjZs2KB3GKU7vg7mdYZVk7Wb0S5VVTJXVJeLYibBY+DiHvjjVTi5Hh6crw2rs0Dvv/8+AL1799Y5kmJkpsCfr0NEGNQOgAf+a1X3JxTTUl0uivlICfu/gfWvgp0jDJkLrQfrHdUdoqOjAWjQoIHOkdwm9iR8P1Sry9L1Oej5Gjg46x2VYmbqpqhiGYSAkHHgdw/89oLFLpRhcYlcSu21q9YAarTSFqTw7aJ3VIoFUgldMT/vJvD46n9+/v1lcPXSWp2OrvrFlW/9+vUA9OvXT99A8vLg4FII/woe/01bDnD0j/rGpFg0dVNU0VdenrYS0uaPYF4n7WafTt2ABWbOnMnMmTN1jYHLB7RJQr9O1UavpMXrG49iFVQfumIZzm6GdS9B3Anw7QqDZoNPM11CuXr1KgC1a9c2/8mzUmHNc3D4J21ceZ93tZWF7FTbS9GoPnTF8jXuAU/t0G6abp/9T9dLThY4OJk1FN0SuZM7OLppn1ju+ZfWBaXqsCjloFroiuXJywU7e+37bwaDSxW4dzrUCTLL6desWQPAoEGDTH+y5BjY/pnWIp+2Dzxq/nMTVFGKoVroinUpSOZ5udCwE+xeAMfWaHXXOz0NzfqatAvik08+AUyc0GMiYPfncPSX/KJmo/+5d6CSuVJBqoWuWL70JG0izd6FcOOStvxdu8dMdrq4uDgAfHxMVK0wKRo+CwBnT+06Ok7RhiQqigFKa6GrhK5Yj9xsiPpVW7Xe1Utb9u74WvAfCs37WebU99xsOLcFjvys1V155Cvt8eNrtU8czp76xqdYHdXlotgGe0cIeOSfn3My4dJ+OLEOHFyhWR9o0R+CR93VaVatWgXAww8/XPGDnN+ujSE/vg7SE8C5CrR56J/+8ZYD7ypGRSmOQQldCNEPmA3YA19KKWfe9nwP4FfgXP5Dq6SU7xovTEUpRvsntFrr0bvhyCo4/hvcvPpPQt85F6rUgQadtFmpBvZNz5kzByhHQk9LgJhwuLgLuj6rfXq4uBui1mh1yds8DE3us+pl+RTrUGaXixDCHjgJ9AFigH3ASCllVJFtegAvSikfMPTEqstFMTopteTq7q0t9DCrKaQnas+5eUMtf2g7BgIf1baNPaGNKnGpdstN1uTkZACqVq36z3Ezb2jDCd2qawn7yiHY9AFcPQI3YrTt7BzgsdXg11UbhmjvDPbqQ7BiXHfb5dIBOC2lPJt/sGXAECCq1L0UxdyE0JI5aIn0xVNw9TDE7NP+vXZUu6kKWnKe1zF/v/x1Nx1coftLVA0dD/Fn4L+dtW6djCTIy9G2HTwX2o3VRuIkXgDfzlCrDdQLhXoh2vR80MaUK4qZGZLQ6wHRRX6OAToWs11nIcRB4DJaa/3o7RsIISYBkwAaNmxY/mgVpTzsHaFeO+3rdo6uMPQrLbGnxWst+Zx0qNaA5cuXQ3oSwxv31CY1uXqBa3Wtld+wk7Z/rTbw9G7zXo+ilMGQhF5cx+Pt/TT7AV8pZYoQYgDwC3DHvG0p5UJgIWhdLuULVVGMyNnj1husRcx/sgcAw61pTVFFwbCEHgMUHSRbH60VXkhKeaPI9+uEEPOEED5SyjjjhKko5rNu3Tq9Q1CUCjFkut0+oJkQopEQwgkYAawuuoEQorYQ2hACIUSH/OOq8nCKVXJzc8PNzU3vMBSl3MpsoUspc4QQ04A/0IYtfi2lPCqEmJL//ALgEeApIUQOkA6MkHrNWFKUu7RkyRIAxowZo3MkilI+aqaootymR48eAGxWfeiKBVIzRRWlHP766y+9Q1CUClEJXVFu4+joqHcIilIhahkURblNWFgYYWFheoehKOWmErqi3EYldMVa6XZTVAgRC1yo4O4+QGUb466uuXJQ11w53M01+0opaxT3hG4J/W4IIcJLustrq9Q1Vw7qmisHU12z6nJRFEWxESqhK4qi2AhrTegL9Q5AB+qaKwd1zZWDSa7ZKvvQFUVRlDtZawtdURRFuY1K6IqiKDbCohO6EKKfEOKEEOK0EOKVYp4XQog5+c8fEkIUszSNdTHgmkfnX+shIcROIUSQHnEaU1nXXGS79kKIXCFE8StTWBFDrlkI0UMIESmEOCqE2GLuGI3NgL/tqkKINUKIg/nXPF6POI1FCPG1EOK6EOJICc8bP39JKS3yC61U7xmgMeAEHARa37bNAOB3tFWVOgF79I7bDNfcBfDK/75/ZbjmItv9DawDHtE7bjP8nquhrdvbMP/nmnrHbYZrfg34d/73NYAEwEnv2O/imu8F2gFHSnje6PnLklvohYtTSymzgILFqYsaAnwrNbuBakKIOuYO1IjKvGYp5U4pZf5S9uxGW0HKmhnyewZ4BlgJXDdncCZiyDWPAlZJKS8CSCmt/boNuWYJeOYvluOBltBzzBum8Ugpt6JdQ0mMnr8sOaEXtzh1vQpsY03Kez1PoL3DW7Myr1kIUQ94CFhgxrhMyZDfc3PASwixWQgRIYR4zGzRmYYh1zwXaIW2xOVh4DkpZZ55wtOF0fOXJZfPNWRxakO2sSYGX48QoidaQu9m0ohMz5Br/gx4WUqZm7/SobUz5JodgBCgF+AK7BJC7JZSnjR1cCZiyDXfD0QC9wFNgL+EENtkkTWLbYzR85clJ/QyF6c2cBtrYtD1CCECgS+B/lJKa1+71ZBrDgWW5SdzH2CAECJHSvmLWSI0PkP/tuOklKlAqhBiKxAEWGtCN+SaxwMzpdbBfFoIcQ5oCew1T4hmZ/T8ZcldLmUuTp3/82P5d4s7AclSyivmDtSIDFmQuyGwChhrxa21osq8ZillIymln5TSD1gBTLXiZA6G/W3/CtwjhHAQQrgBHYFjZo7TmAy55oton0gQQtQCWgBnzRqleRk9f1lsC10atjj1OrQ7xaeBNLR3eKtl4DW/BXgD8/JbrDnSiivVGXjNNsWQa5ZSHhNCrAcOAXnAl1LKYoe/WQMDf8/vAWFCiMNo3REvSymttqyuEGIp0APwEULEAG8DjmC6/KWm/iuKotgIS+5yURRFUcpBJXRFURQboRK6oiiKjVAJXVEUxUaohK4oimIjVEJXTC6/QmKkEOKIEOKn/HHVhu47Tggxt5znSynh8XeFEL3zv98shAjN/36dEKJa/tfU8pyrjDg+zq8a+PFtj48TQsTmvybHhRAvGHCswtgVpSRq2KJickKIFCmlR/733wMRUsr/FnneXkqZW8K+44BQKeW0ipyvlG02Ay9KKcOLPOYH/Cal9Df0XGWc4wZQQ0qZedvj48i/JiGEN3ACaCuljC7mMIpiMNVCV8xtG9A0v9b3JiHED8BhIYSLEGKxEOKwEOJAfq2aAg2EEOvza2m/XfCgEOKX/MJVR4UQk4qeRAjxiRBivxBioxCiRv5jYaKYWupCiPNCCB9gJtAkv+X8sRDiOyHEkCLbfS+EGHzbviJ/2yP5sQ/Pf3w14A7sKXisOPmlG04DdfL3e0sIsS//eAtF/uyxorHnx/tO/vUdFkK0NOB1VyoBldAVsxFCOKDVcD+c/1AH4HUpZWvgaQApZQAwEvhGCOFSZLvRQDAwrKCrBJggpQxBq/XybH5rF7REul9K2Q7YgjZDzxCvAGeklMFSyulo9XLG58deFa0W/brb9nk4P64goDfwsRCijpRyMJCef6zlpbwmDQEXtBmhAHOllO3zPyW4Ag+UsGtc/vXNB1408PoUG6cSumIOrkKISCAcrV7HV/mP75VSnsv/vhvwHYCU8jhwAa2ELMBfUsp4KWU6Wh2bggqTzwohDqLVhW8ANMt/PA8oSKJLqGBFSinlFrRPEzXR3mRWSilvr8/dDVgqpcyVUl5DewNpb8DhhwshjqLVKpktpczIf7ynEGJP/vT3+4A2Jey/Kv/fCMDP4ItSbJrF1nJRbEq6lDK46AP5PQmpRR8qZf/bb/RIIUQPtBZxZyllWn6fuAvFu5sbRd+hfToYAUwo5vmK1vNdnt+H3hlYK4T4HUgC5qH1r0cLIWZQ8jUV9Mvnov4fK/lUC12xFFvREidCiOZAQ7SbhQB9hBDVhRCuwIPADqAqkJifzFuiLeFVwA4o6CsfBWw3MIabgOdtj4UBzwNIKY+WEPdwIYR9fl/9vZSj3KuUchfam8Zz/JO844QQHvxzDYpiEPXOrliKecCC/K6GHGCclDIzvyW/HS3pNQV+kFKG5283RQhxCC3x7y5yrFSgjRAiAkgGSrwpWZSUMl4IsUNoi/r+LqWcLqW8JoQ4BvxSwm4/A53R1siUwEtSyqvlunL4N7Af+BBYhHaP4TxayVlFMZgatqgopcgfM38YaCelTNY7HkUpjepyUZQS5E/kOQ78TyVzxRqoFrqiKIqNUC10RVEUG6ESuqIoio1QCV1RFMVGqISuKIpiI1RCVxRFsRH/D//Z3rKm2Yc0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "prob_rain = np.arange(0, 1.001, 0.01)\n",
    "vectors = [np.array([1 - p, p]) for p in prob_rain]\n",
    "magnitudes = [np.linalg.norm(v) for v in vectors] \n",
    "square_magnitudes = [v @ v for v in vectors]\n",
    "plt.plot(prob_rain, magnitudes, label='Magnitude')\n",
    "plt.plot(prob_rain, square_magnitudes, label='Squared Magnitude',\n",
    "         linestyle='--')\n",
    "plt.xlabel('Probability of Rain')\n",
    "plt.axvline(0.5, color='k', label='Perfect Balance', linestyle=':')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`v @ v` serves as an excellent metric for class-label imbalance. However, data scientists prefer the slightly different metric of `1 - v @ v`. This metric, called **Gini Impurity**, essentially flips the plotted curve.\n",
    "\n",
    "**Listing 22. 32. Plotting the Gini Impurity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAw3UlEQVR4nO3deXxU1f3/8dcnCVmAEAIkLCELgbBvQtgUQVzBDXeEuuBO61JrbdW2P21rW23tYt3qXhdU3C0qaK0KKmvCvkOAbKwhgQSyL5/fHzP4TdMQhpCbm8l8no/HfZCZuTPzPhDmM+fec88RVcUYY0zgCnI7gDHGGHdZITDGmABnhcAYYwKcFQJjjAlwVgiMMSbAhbgd4ER16dJFk5KS3I5hjDF+ZcWKFQdUNaa+x/yuECQlJZGenu52DGOM8SsiknWsx+zQkDHGBDgrBMYYE+CsEBhjTICzQmCMMQHOCoExxgQ4RwuBiEwWkS0ikiEi99fz+BkiUigiq73bg07mMcYY878cGz4qIsHA08A5QC6QJiJzVXVjnV2/VdULncphjDGmYU5eRzAayFDVHQAiMgeYCtQtBMb4BVWlsLSS3IOlFBRXcLCkgkMllZRVVlNZXUNFtRISJISGBBEWEkRURBui24YS3S6UHh3DiWkfhoi43Qxj/oeThSAOyKl1OxcYU89+40RkDbAbuFdVN9TdQURuBW4FSEhIcCCqMf+tqKySNTmH2LSniM17DrNl32GyC0o4XFbV6NcMbxNEfHRbUrq2p3+3DvTvFsnw+I7EdghvwuTGnDgnC0F9X33qroKzEkhU1SMicj7wEZDyP09SfR54HiA1NdVW0jFN7kh5FYszDvBdxgHSMg+yeW8RR9ds6tYhnL7dIklNjCa+U1t6RkfQpX0Y0e1C6RjRhvA2wYSGBBESJFTXKJXVSnlVNYWllRQUV5B/pILdhaVk55eQXVDCxt1FzF+/9/vXT+zcltTETpzauzMT+8XQpX2Ye38RJiA5WQhygfhat3vi+db/PVUtqvXzPBF5RkS6qOoBB3MZA8D+w2XMX7eXzzfsJS2zgMpqpW1oMCMTo7n7rL6MTIxmYI8OdGoX6vNrhgQLIcEQERpMx7ahJHZuV+9+xeVVbN5bxKrsQ6RlFrBgy37eX5kLwJC4KM4Z2JULh3YnOaZ9k7TVmIaIU0tVikgIsBU4C9gFpAEzah/6EZFuwD5VVREZDbyHp4dwzFCpqalqcw2Zxiour+LTdXv4YGUuy3YWoAopse05c0AsZ/SNZWRiNKEhzT+quqZG2biniAVb9vP1ljxWZh9EFQZ078Alw3tw2YiexERaT8E0noisUNXUeh9zcs1i7+Gex4Fg4GVV/b2IzAJQ1WdF5A7gh0AVUArco6qLG3pNKwSmMTbsLmT20izmrt5NcUU1yV3aceGwHlw0tDspXSPdjvc/9haWMW/dHj5eu5tV2YcICRLOGhDLjDGJTEjpYiedzQlzrRA4wQqB8VVNjfLl5v289N0Olu4oIKJNMBcM7c7Vo+IZmRjtNx+mGfsP8056Lu+vyCW/uIKU2PbcNL4Xl5wSR3ibYLfjGT9hhcAElOoa5ZO1u3nyqwwy9h+hR1Q4M09LYtqoBKIi2rgdr9HKq6r5ZM0eXvpuJxv3FNGlfRizJiYzY0wCbUP9bkZ508ysEJiAUFOjfLJuD49/sZUdB4rp27U9t0/qwwVDuhMS3HpmU1FVluzI5+mvM1iUkU/ndqHMmtiba8clWg/BHJMVAtPqLd5+gEfmbWbdrkL6d4vkx2elcN6gbgQF+cfhn8ZKzyzg719u49ttB4jrGMG95/Vl6rC4Vt9uc+KsEJhWKzu/hN9+soH/bNpPj6hw7j2vH5cMD7wPwsUZB/jD/E2s31XE4LgO/ObiwYxMjHY7lmlBrBCYVqessppnFmzn2YXbCQkS7jwzhRtOSwroQyM1NcrHa3fzyLzN7C0q48qRPblvSn+7QM0ADRcCO8Nk/M6S7fnc/8FasvJLuHhYD35x/gC6Rdk0DUFBwtThcZw9oCtPfLWNl7/byb837uNXFwzgipE9/WaUlGl+1iMwfqOorJJH5m3mreXZJHZuyyOXDuHUPl3cjtViZew/wgMfrCUt8yCnp3ThD5cOIb5TW7djGZfYoSHj95Zsz+en76xmb1EZN5+ezE/O7ktEaOAeBvJVTY0ye1kWf5y/GYCHLh7EldY7CEgNFYLWM6bOtErlVdU8Mm8TM15cSlibYN7/4an84vwBVgR8FBQkXDcuic9/MoHBcVH8/L21/HD2SgqKK9yOZloQKwSmxco8UMxlzyzmuW92MH10Ap/eNZ5TEmwkTGP0jG7Lm7eM5YEp/fly8z4mP/4NS3fkux3LtBBWCEyL9OnaPVz45HfkHizlhetS+cOlQ+zq2ZMUHCTcNrE3H91+Gu3CQpjxwlKe/jqDmhr/Ojxsmp4VAtOiVFbX8Ou5G7j9zZX0iW3Pp3eN55yBXd2O1aoM6hHFx3eO54KhPXjs8y3c8Eoah0rsUFEgs0JgWoz8I+Vc8+IyXlmcyY2n9eKd28bRM9pGuTihfVgIT1w9nN9dMpjF2w8w9elFbNl72O1YxiVWCEyLsH5XIRc/tYjVOYd4fNpwHrxooCvrAgQSEeGasYnMuXUcpRXVXPrMIj5bv8ftWMYF9j/NuO7zDXu54tnFqCrvzTqVS06JcztSQBmZGM3Hd46nb9dIZs1eydNfZ+Bvw8rNybFCYFyjqrzwzQ5mzV5B/24d+Ncd4xnSM8rtWAGpa4dw5tw6lqnDPecN7nt/LRVVNW7HMs3EhmEYV1TXKA/NXc/spdmcP6Qbf71qeEDPE9QShLcJ5vFpw0ns3I4nvtxGTkEpz103kg7h/ruGg/GN9QhMsyurrOZHb6xg9tJsbpuYzFPTR1gRaCFEhHvO6ctfrhxGWmYB055byv6iMrdjGYdZITDNqqiskutfXs7nG/bx4IUDeWDKgICbMtofXD6yJy/NHEVWfjGXP7uYzAPFbkcyDrJCYJrNgSPlTHtuKSuyDvL3q4dz4/hebkcyDZjYN4Y3bxnLkbIqrnh2MRt3F7kdyTjECoFpFvuKypj23BJ2HjjCi9enMnW4jQzyB8PjO/LeD0+lTXAQ019YypqcQ25HMg6wQmAcl3uwhKueW8LewjJeuWE0Z/SLdTuSOQG9Y9rzzm3j6BARwg9eXEZaZoHbkUwTs0JgHJVTUMK055ZysLiC128ew9jkzm5HMo0Q36kt79w2jtjIMK57ablNWNfKWCEwjsk9WML0F5ZypLyKN28ZywibOdSvdY+KYM5tY4mLjuDGV9KsZ9CKWCEwjth9qJTpLyylsLSS2TeNYXCcXSjWGsRGhvPmLWPoFhXOzJeXsyLLikFrYIXANLn9RWXMeGEph4o9RcCuFm5dYiPDeeuWscR2COf6l9PsBHIrYIXANKmDxRVc89Iy9h8u59WbRjMsvqPbkYwDunbwFIPodm24/p/L2brPZi71Z1YITJM5Ul7FzFfSyMwv4cXrUu2cQCvXLSqcN24aS2hwENe8uIzs/BK3I5lGskJgmkRZZTW3vJrO+l2FPDX9FE7t08XtSKYZJHRuy+ybx1BRXcOMF5eyz6aj8EtWCMxJq65R7nlnNUt25PPYFUM5d1A3tyOZZtS3aySv3jCag8UVXP/ycorKKt2OZE6QFQJzUlSVhz/ZyLx1e/nl+QO4bERPtyMZFwyL78g/rhlJxv4j3PbaCsqrqt2OZE6Ao4VARCaLyBYRyRCR+xvYb5SIVIvIFU7mMU3v2YU7eGVxJjeN78UtE5LdjmNcNKFvDH+6YihLduTz03fWUFNji9v4C8fWIxCRYOBp4BwgF0gTkbmqurGe/f4IfO5UFuOMuWt288fPNnPRsB788vwBbscxLcBlI3qy/3A5j87fTFx0BA9Msd8Lf+Bkj2A0kKGqO1S1ApgDTK1nvzuB94H9DmYxTWxFVgH3vruG0Umd+POVQ20qafO92yYkc83YBJ5buIO3lme7Hcf4wMlCEAfk1Lqd673veyISB1wKPNvQC4nIrSKSLiLpeXl5TR7UnJis/GJueW0FcR0jeO7akYSF2KIy5v+ICL++aBAT+8bwq4/W8922A25HMsfhZCGo7yti3YOGjwP3qWqDZ5ZU9XlVTVXV1JiYmKbKZxqhsLSSG19Jo0aVl2eOIrpdqNuRTAsUEhzEUzNOISW2PT+cvYJtdsFZi+ZkIcgF4mvd7gnsrrNPKjBHRDKBK4BnROQSBzOZk1Bdo9z51iqyC0p47pqR9OrSzu1IpgWLDG/DSzNHEdYmmFteS+dQSYXbkcwxOFkI0oAUEeklIqHA1cDc2juoai9VTVLVJOA94Eeq+pGDmcxJeHT+Jr7Zmsdvpw5mjE0nbXzgOXw4gt2HyrjjzVVUVde4HcnUw7FCoKpVwB14RgNtAt5R1Q0iMktEZjn1vsYZ76/I5YVvd3L9uESmj05wO47xIyMTO/G7SwbzXcYBfj9vk9txTD0cGz4KoKrzgHl17qv3xLCqznQyi2m8NTmHeODDdYxL7syvLhzodhzjh64aFc+mvUX8c1EmA7t34MrU+OM/yTQbu7LYNCj/SDk/nL2CmPZhPPODEbQJtl8Z0zi/PH8Ap/buzC8/Ws/6XYVuxzG12P9qc0xV1TXc+dYq8osreO7akTZCyJyUkOAgnpx+Cl3ahXLb6ys4WGwnj1sKKwTmmB779xYWb8/nd5cMthXGTJPo3D6Mf1wzkrzD5dw1ZxXVNg1Fi2CFwNTrs/V7eW7hDn4wJsGO55omNSy+I7+ZOohvtx3g7//Z6nYcgxUCU4/s/BJ+9t4ahvaM4sGL7OSwaXrTRydwxciePPl1Bt9stdkC3GaFwPyX8qpqbn9zJQI8PWOETR9hHPPw1MH0jY3k7rdXs7fQFrRxkxUC819+/+km1u0q5M9XDiO+U1u345hWLCI0mKd/MIKyymruessuNnOTFQLzvfnr9vDakixuHt/LVhkzzaJPbHv+cOkQlmcW8Dc7X+AaKwQGgF2HSrnv/bUMi+/Izyf3dzuOCSCXnBLHtNR4nlmwncUZNlOpG6wQGKqqa7h7zipqFJ64ejihIfZrYZrXQxcPpFeXdtz99moK7PqCZmf/4w1PfpVBWuZBfn/pYBI724yipvm1DQ3hyemncKikkp+9uwZVu76gOVkhCHBpmQU8+dU2Lh/Rk6nD447/BGMcMqhHFA+c358vN+/n1cWZbscJKFYIAtjhskrunrOa+E5t+c3UQW7HMYaZpyYxqV8Mj8zfbIvZNCMrBAHs13M3sqewlL9eNZz2YY5ORGuMT0SEP14xlHZhIdz99moqqmxIaXOwQhCg5q3bw/src7ljUh9GJka7HceY78VGhvPIZUPYsLuIx21IabOwQhCA9hWV8YsP1zGsZxR3npXidhxj/sd5g7pxVWpP/rFwO8t3Frgdp9WzQhBgVJX73l9LWWU1f5023NYXMC3WgxcNIj66LT99dzXF5VVux2nV7FMgwLyTnsOCLXncN7k/vWPaux3HmGNqHxbCY1cMJfdgKY/MtyUunWSFIIDsOlTKw59sYmxyJ64fl+R2HGOOa0xyZ244tRezl2bz3Ta76tgpVggChKpy33trUVUeu2IYQUHidiRjfPLzyf1I7tKO+95fy+GySrfjtEpWCALEG8uy+S7jAL+4YIDNKmr8SnibYP581TD2FJby+0/tEJETrBAEgN2HSnl0/mZO69OZGaMT3I5jzAkbkRDNLacnMycth0U2MV2TO24hEJE7RMQGmvspVeUXH66jukZ59LKhiNghIeOffnJOX3p1acf9H6ylpMJGETUlX3oE3YA0EXlHRCaLfZL4lQ9X7WLBljx+PrmfHRIyfi28TTCPXjaEnIJSHvt8i9txWpXjFgJV/RWQArwEzAS2icgfRKS3w9nMSco7XM5vP9nIyMRoGyVkWoUxyZ25dmwiryzOZEWWXWjWVHw6R6CeOWH3ercqIBp4T0T+5GA2c5J+8/EGSsqr+ePlQ22UkGk17pvSnx5REdz//jqbi6iJ+HKO4C4RWQH8CVgEDFHVHwIjgcsdzmca6avN+/hk7R7uOLMPfWLtwjHTerQPC+HhSwaxbf8Rnlu43e04rYIvPYIuwGWqep6qvquqlQCqWgNc6Gg60yjF5VX8v482kBLbnlkT7QieaX3O7N+VC4Z258mvM9iRd8TtOH7Pl0LQS1Wzat8hIq8DqKoN6m2B/vbFVnYdKuWRy4bYspOm1XroooGEhQTxiw/X2YpmJ8mXT4n/WrFERILxHBYyLdC63EJeXrSTGWMSSE3q5HYcYxwTGxnOA1MGsHRHAe+m57odx68dsxCIyAMichgYKiJF3u0wsB/4ly8v7h1uukVEMkTk/noenyoia0VktYiki8j4RrfEUF2j/PKjdXRuH8Z9k/u7HccYx109Kp5RSdH8Yf4mW/T+JByzEKjqI6oaCTymqh28W6SqdlbVB473wt6ew9PAFGAgMF1EBtbZ7UtgmKoOB24EXmxsQwy8uTybtbmF/OqCAURFtHE7jjGOCwoSfnfJEI6UVfHH+ZvdjuO3GuoRHP1K+a6IjKi7+fDao4EMVd2hqhXAHGBq7R1U9Yj+38G9doAd6GukvMPl/OkzzzQSFw/r4XYcY5pNv26R3DS+F2+n59i1BY3U0EK19wC3An+p5zEFzjzOa8cBObVu5wJj6u4kIpcCjwCxwAX1vZCI3OrNQkKCzZVTn0fmbaKssprfTh1s00iYgHPXWSl8vGY3v/xwPZ/cOZ4QW3DphDR0aOhWEQkCfqWqk+psxysCAPV9Gv3PN35V/VBV+wOXAA8fI8vzqpqqqqkxMTE+vHVgWbojnw9W7eK2Cb1tsRkTkNqFhfDgRYPYvPcwryzOdDuO32mwbHqvFfhzI187F4ivdbsnsLuB9/oG6C0iXRr5fgGpsrqGh/61gZ7REdw+qY/bcYxxzXmDujKpXwyP/2cb+4vK3I7jV3zpP/1bRC5vxGRzaUCKiPQSkVDgamBu7R1EpM/R1/WedwgF8k/wfQLa60uy2LLvMP/vwoFEhAa7HccY14gID140iIqqGh61E8cnxJdCcA/wLlB+dAipiBQd70mqWgXcAXwObALeUdUNIjJLRGZ5d7scWC8iq/GMMJqmdmWIz/IOl/O3L7YyoW8M5w7s6nYcY1zXq0s7bj69Fx+s2kV6pp049pX42+duamqqpqenux2jRfjZu2v4aPUuPrt7gp0bMMarpKKKs/6ykOi2oXx853iCbcJFAERkhaqm1veYL5POTahva/qY5kSszD7IuytyuXF8LysCxtTSNjSEX5w/gI17inhzebbbcfxCQ8NHj/pZrZ/D8VwfsILjDx81DqmpUX49dwOxkWHceWaK23GMaXEuHNqdN5Zl8Zd/b+Giod3p2DbU7Ugtmi8L01xUazsHGAzscz6aOZYPVu1ibW4h90/pT/swX2q5MYFFRHjookEUlVby+H+2uR2nxWvMVRe5eIqBcUFxeRV/+mwzw+I7csnwOLfjGNNiDejegatHJ/D60iwy9h92O06L5ss5gidF5Anv9hTwLbDG+WimPs8syGD/4XIeumigrTpmzHH89Jy+tA0N5uFPbMb8hvjSI0jHc05gBbAEuE9Vr3E0lalXTkEJL3y7k0uG92BEQrTbcYxp8Tq3D+PHZ6WwcGseX2/e73acFsuXcwSvAm8Bq4C1eC4UMy54dP5mgkW4b4pNMW2Mr64bl0Ryl3Y8/OlGKqttjeP6+HJo6HxgO/AE8BSQISJTnA5m/tuKrAI+XbeHWyck0z0qwu04xviN0JAgHjh/ADvyinnLhpPWy5dDQ38FJqnqGao6EZgE/M3ZWKY2VeV3n24iNjKM2yYmux3HGL9z9oBYxiZ34vH/bKOorNLtOC2OL4Vgv6pm1Lq9A88qZaaZfLJ2D6uyD3Hvuf1oG2rDRY05USLCry4YyMGSCp7+OuP4TwgwvhSCDSIyT0Rmisj1wMdAmohcJiKXOZwv4JVXVfPHzzbTv1skl4/s6XYcY/zW4LgoLj0ljn8uyiSnoMTtOC2KL4UgHM8FZBOBM4A8oBNwEXChY8kMAK8uziT3YCm/vGCAzZlizEm699x+CPDY51vcjtKiHPc4g6re0BxBzP86VFLBU19lMLFvDKen2II8xpysHh0juOX0ZJ76OoObT+/F0J4d3Y7UIvgyaqiXiPxVRD4QkblHt+YIF+ieWbCdw+VV3G/DRY1pMrdNTKZTu1Aenb8Zf5t92Sm+nHn8CHgJz7kBG4TbTHYdKuWVxZlcdkpPBnTv4HYcY1qNyPA23HlmH37z8UYWbs3jjH6xbkdynS+FoExVn3A8ifkvf/33VgDuObevy0mMaX1mjEng5UU7eXT+ZiakxAT8dC2+nCz+u4g8JCLjRGTE0c3xZAFs054iPliVy8xTk4jraBePGdPUwkKCuffcfmzee5iPVu9yO47rfOkRDAGuxbP+wNFDQ4qtR+CYP322mciwEH50Rm+3oxjTal00tAcvfruTv/x7KxcM7U5YSOCu+e1Lj+BSIFlVJ6rqJO9mRcAhaZkFfL0ljx+e0ccW0zDGQUFBws8n92PXoVLeXBbYU0/4UgjWAB0dzmHwTCXxp882ExsZxsxTk9yOY0yrN75PF8Yld+aprzIoLq9yO45rfCkEXYHNIvK5DR911oIteaRlHuTOs1KICA3cbqoxzUVE+NnkfuQXV/DydzvdjuMaX84RPOR4CkNNjfKnz7eQ0Kkt01Lj3Y5jTMAYkRDNOQO78vw3O7hmbCLR7QLvkKwv6xEsrG9rjnCB5JN1e9i0p4h7zulLaEhjVhA1xjTWz87rx5GKKp5duN3tKK445ieOiBwWkaJ6tsMiUtScIVu7quoa/vbFVvp3i+TiYT3cjmNMwOnbNZJLT4njlcWZ7CsqcztOsztmIVDVSFXtUM8Wqap2qWsT+mDVLnYeKOaec/oG/IUtxrjl7rP6Ul2jPBOA01TbMQiXVVTV8MSX2xjaM4pzBnZ1O44xASuhc1uuTI3nreU57DpU6nacZmWFwGXvpOeQe7CUe87pi4j1Boxx051n9gHgqa+2uZykeVkhcFFZZTVPfZXByMRoJva1aaaNcVuPjhHMGJPAu+m5ZOcHzuI1Vghc9NbybPYWlfHTc603YExL8aMzehMcJPz9y8DpFTQ0aug77591Rw/ZqKEmUFZZzTMLtjMuuTOn9u7idhxjjFdsh3CuG5fIh6ty2ZF3xO04zaKhUUPjvX/WHT1ko4aawBvLssk7XM7dZ6e4HcUYU8dtE3sTGhLEU18Fxgginw4NiUiwiPQQkYSjm4/PmywiW0QkQ0Tur+fxH4jIWu+2WESGnWgD/FFZZTXPLtzOqb07Mya5s9txjDF1dGkfxnXjkvho9a6A6BX4slTlnXgWr/8C+NS7feLD84KBp4EpwEBguogMrLPbTmCiqg4FHgaeP6H0fupob+DHZ1lvwJiW6tYJyQHTK/ClR/BjoJ+qDlLVId5tqA/PGw1kqOoOVa0A5gBTa++gqotV9aD35lKg54mE90fWGzDGPwRSr8CXQpADFDbiteO8zz0q13vfsdwEzK/vARG5VUTSRSQ9Ly+vEVFaDusNGOM/AqVX4Esh2AEsEJEHROSeo5sPz6tvPKTWu6PIJDyF4L76HlfV51U1VVVTY2L8d7z90d7AuGTrDRjjD2r3CnYeKHY7jmN8KQTZeM4PhAKRtbbjyQVqz6fcE9hddycRGQq8CExV1XwfXtdvvZ2WQ97hcu6y3oAxfuPm03vRJjioVc9BdNz1CFT1N4187TQgRUR6AbuAq4EZtXfwjj76ALhWVbc28n38QnmVpzcwKimascmd3I5jjPFRbGQ400cnMHtpFnedlUJ8p7ZuR2pyDV1Q9rj3z49rr0zm6wplqloF3AF8DmwC3lHVDSIyS0RmeXd7EOgMPCMiq0Uk/WQb1FJ9sHIXewrLuPPMFLuK2Bg/M2tib4JEWu16BQ31CF73/vnnxr64qs4D5tW579laP98M3NzY1/cXldU1PLMgg2HxHTk9xa4iNsbfdIsK58rUnrybnsudZ6bQLSrc7UhNqqEri1d4/7QVyk7Sv1bvJqeglDsn9bHegDF+atbE3tSotspeQUOHhqaKyO21bi8TkR3e7Yrmief/ji50MbB7B84aEOt2HGNMI8V3asulp8Tx1nLPEPDWpKFRQz8Hap8LCANGAWcAP3QwU6vy2fq97DhQzO3WGzDG7/1oUh8qq2t4edFOt6M0qYYKQaiq1r4g7DtVzVfVbKCdw7laBVXl6a8zSO7SjsmDu7kdxxhzknp1aceUId15fUkWhaWVbsdpMg0VgujaN1T1jlo3/feqrma0cGseG/cUMcs7v7kxxv/96IzeHCmvYvbSLLejNJmGCsEyEbml7p0ichuw3LlIrcczX2+nR1Q4lwxvaGYNY4w/GdQjikn9Ynjpu52UVlS7HadJNFQIfgLcICJfi8hfvNsCYCZwdzNk82tpmQUszyz4fq4SY0zrcfukPhQUVzAnLdvtKE2ioeGj+1X1VDzTQ2d6t9+q6jhV3dc88fzXM19n0LldKNNG+bR0gzHGj6QmdWJ0Uide+GYHFVU1bsc5acf9qqqqX6nqk97tq+YI5e827Sni6y153HBaEhGhwW7HMcY44EeTerO7sIy5a/5nCjW/Y8csHPDcwu20Cw3m2rFJbkcxxjhkYt8Y+neL5LmF26mpqXdiZb9hhaCJ5RSU8PHaPUwfnUBU2zZuxzHGOEREmDWxN9v2H+GrzfvdjnNSrBA0sZe+24kAN53ey+0oxhiHXTC0O3EdI3juG/+edsIKQRM6WFzB22k5TB0eR/eoCLfjGGMc1iY4iJtP70Va5kFWZBW4HafRrBA0oVeXZFJaWc2sicluRzHGNJNpo+KJbtuGfyzY4XaURrNC0ERKK6p5dXEmZw+IJaWrLwu4GWNag7ahIVw3Lon/bNpHxv7DbsdpFCsETeS9lbkcLKnkltOtN2BMoLluXCJhIUG8+K1/TkZnhaAJVNcoL327g2HxHRndy5ahNCbQdG4fxhUje/LByl3sP1zmdpwTZoWgCXyxcR+Z+SXcenqyTTVtTIC6aXwvKmtqeH2J/01GZ4WgCbzw7Q7iO0Vw3qCubkcxxrgkOaY95wzoyutLsyipqHI7zgmxQnCSVmQVsCLrIDed1ouQYPvrNCaQ3TohmUMllby3ItftKCfEPrlO0gvf7CQqog1Xpsa7HcUY47KRidGcktCRF7/dSbUfTTthheAkZOUX8/nGvVwzNoF2YSFuxzHGuExEuPX0ZLILSvhi41634/jMCsFJ+OeiTEKChOvHJbkdxRjTQpw7qBvxnSJ46Tv/GUpqhaCRCksreSc9h4uG9iC2Q7jbcYwxLURwkDDzVM+0E2tyDrkdxydWCBrp7bRsSiqquXG8TS5njPlvV6X2pH1YiN/0CqwQNEJVdQ2vLMpkbHInBsdFuR3HGNPCRIa34epR8cxbt4fdh0rdjnNcVggaYf76vewuLOOm8TadhDGmftefmkSNKq8uyXQ7ynFZIWiEl77bSVLntpzVP9btKMaYFiq+U1smD+7GW8uyKS5v2ReYWSE4QSuzD7I65xA3nNaLoCCbTsIYc2w3je9FUVkVH6xs2ReYWSE4Qf9clElkWAiXj+zpdhRjTAs3IiGaoT2j+OfizBa9rrGjhUBEJovIFhHJEJH763m8v4gsEZFyEbnXySxNYW9hGfPX7eGqUfG0twvIjDHHISLccFoSO/KK+WZbnttxjsmxQiAiwcDTwBRgIDBdRAbW2a0AuAv4s1M5mtLspVlUq9oFZMYYn10wpAcxkWG8sjjT7SjH5GSPYDSQoao7VLUCmANMrb2Dqu5X1TSg0sEcTaKsspo3l2dz9oCuJHRu63YcY4yfCA0J4poxiSzYksf2vCNux6mXk4UgDsipdTvXe98JE5FbRSRdRNLz8tzpXs1dvZuC4gpuOC3Jlfc3xvivGWMSCA0O4tUW2itwshDUN6SmUWdLVPV5VU1V1dSYmJiTjNWo9+flRTvp1zWSccmdm/39jTH+LSYyjAuHdee9FbkUlra8AyBOFoJcoPbczD2B3Q6+n2OW7yxg897D3HBakq1AZoxplBtP60VJRTXvpuccf+dm5mQhSANSRKSXiIQCVwNzHXw/x7y2JIuoiDZMHd6oI1vGGMPguChGJkYze2lWixtK6lghUNUq4A7gc2AT8I6qbhCRWSIyC0BEuolILnAP8CsRyRWRDk5laoy9hWV8tmEv00bFExEa7HYcY4wfu25cIpn5JSxsYUNJHR0Mr6rzgHl17nu21s978RwyarHeXJZFjSrXjEl0O4oxxs9NGdyd30Vu4rXFmUzq13KmqLErixtQXuUZMnpW/1gbMmqMOWmhIUHMGJ3Agq15ZB4odjvO96wQNGD+ur0cOFLBdXYBmTGmicwYk0CwCLOXZrkd5XtWCBrw6pJMkru0Y3yfLm5HMca0El07hDN5cDfeSc+hpKJlzEpqheAY1u8qZFX2Ia4Zm2izjBpjmtT1pyZRVFbFR6taxoh6KwTH8PqSLCLaBHNFaos+l22M8UOpidH07xbJ7KVZqLo/lNQKQT0KSyr515pdXHJKHB3C27gdxxjTyogI145LZOOeIlZmH3I7jhWC+ry7IoeyyhquGZvgdhRjTCt1yfA42oeFtIiTxlYI6qipUd5Yls3IxGgG9bCF6Y0xzmgXFsLlI+L4dO0e8o+Uu5rFCkEdi7YfYOeBYq4daxeQGWOcdc3YRCqqa3gn3d2lLK0Q1PH6kiw6tQtlypBubkcxxrRyKV0jGZvciTeWZVHt4vxDVghq2X2olP9s2se0UfGEhdi8QsYY5107Noncg6Us3LrftQxWCGqZk5aDAjNG20liY0zzOHdQV2Iiw3hjabZrGawQeFVV1/B2WjYT+8YQ38nmFTLGNI82wUFMS43n6y372XWo1JUMVgi8vty8n31F5fzAZhk1xjSzq0fHo8Dby93pFVgh8HpjWTbdOoQzqV/zL4VpjAlsPaPbMqlfLHPScqisrmn297dCAGTnl/DN1jyuHh1PSLD9lRhjmt+M0QnsP1zOl5v2Nft726ce8FZaNkEC00bFH39nY4xxwKT+sfSICueNZc1/eCjgC0FFVQ3vpOVw1oCudI+KcDuOMSZABQcJV49O4NttB8jKb95FawK+EPx7417yiyuYMcaGjBpj3DVtVDzBQcKbzXzSOOALwVvLs4nrGMGEFDtJbIxxV9cO4ZzVP5b3V+RSUdV8J40DuhBk5RezKCP/+ypsjDFumz46gQNHKvhPM540DuhCMCcthyCBq1LtJLExpmWY0DeGHlHhvNWMh4cCthBUVtfwbnouZ/bvSreocLfjGGMM4DlpfNWoeL7ddoCcgpJmec+ALQRfbtrHgSPlTB9tvQFjTMtyVWo8QQJz0pqnVxCwheCt5Tl0jwpnYl87SWyMaVl6dIzgjH6xvJue2yxXGgdkIcg9WMI32/K4KtWuJDbGtEzTvVcaf7XZ+empA/JT8OhqQFfZlcTGmBZqUr8YYiPDeDstx/H3CrhCUF2jvJuew4SUGOI62pXExpiWKSQ4iCtTe7Jgy372FpY5+l4BVwi+3ZbHnsIym1fIGNPiXZUaT43Ceyuc7RUEXCF4Oy2HTu1COXtAV7ejGGNMgxI7t2NccmfeTs+hxsE1jQOqEBw4Us5/Nu3jslPiCA0JqKYbY/zUtFHx5BSUsnRHvmPv4einoYhMFpEtIpIhIvfX87iIyBPex9eKyAgn83y4cheV1WqHhYwxfmPy4G50CA9hjoMnjR0rBCISDDwNTAEGAtNFZGCd3aYAKd7tVuAfTuVRVd5Oz2FkYjQpXSOdehtjjGlS4W2CufSUOD7bsJdDJRWOvIeTPYLRQIaq7lDVCmAOMLXOPlOB19RjKdBRRLo7EWZl9kEy9h9hms0rZIzxM1eNiqeiqoaPVu1y5PWdLARxQO2+TK73vhPdBxG5VUTSRSQ9Ly+v0YEm9I3hgqGO1BljjHHMoB5RTB3eg+h2oY68fogjr+pR37zOdU97+7IPqvo88DxAampqo06dj0zsxGs3jm7MU40xxnV/v/oUx17byR5BLlD7OExPYHcj9jHGGOMgJwtBGpAiIr1EJBS4GphbZ5+5wHXe0UNjgUJV3eNgJmOMMXU4dmhIVatE5A7gcyAYeFlVN4jILO/jzwLzgPOBDKAEuMGpPMYYY+rn5DkCVHUeng/72vc9W+tnBW53MoMxxpiG2eW1xhgT4KwQGGNMgLNCYIwxAc4KgTHGBDjxnK/1HyKSB2Q18uldgANNGMcfWJsDg7U5MJxMmxNVtd5F2v2uEJwMEUlX1VS3czQna3NgsDYHBqfabIeGjDEmwFkhMMaYABdoheB5twO4wNocGKzNgcGRNgfUOQJjjDH/K9B6BMYYY+qwQmCMMQGuVRYCEZksIltEJENE7q/ncRGRJ7yPrxWREW7kbEo+tPkH3rauFZHFIjLMjZxN6XhtrrXfKBGpFpErmjOfE3xps4icISKrRWSDiCxs7oxNzYff7SgR+VhE1njb7NezGIvIyyKyX0TWH+Pxpv/8UtVWteGZ8no7kAyEAmuAgXX2OR+Yj2eFtLHAMrdzN0ObTwWivT9PCYQ219rvKzyz4F7hdu5m+HfuCGwEEry3Y93O3Qxt/gXwR+/PMUABEOp29pNo8wRgBLD+GI83+edXa+wRjAYyVHWHqlYAc4CpdfaZCrymHkuBjiLiz4sZH7fNqrpYVQ96by7FsxqcP/Pl3xngTuB9YH9zhnOIL22eAXygqtkAqurv7falzQpEiogA7fEUgqrmjdl0VPUbPG04lib//GqNhSAOyKl1O9d734nu409OtD034flG4c+O22YRiQMuBZ6ldfDl37kvEC0iC0RkhYhc12zpnOFLm58CBuBZ5nYd8GNVrWmeeK5o8s8vRxemcYnUc1/dMbK+7ONPfG6PiEzCUwjGO5rIeb60+XHgPlWt9nxZ9Hu+tDkEGAmcBUQAS0RkqapudTqcQ3xp83nAauBMoDfwhYh8q6pFDmdzS5N/frXGQpALxNe63RPPN4UT3cef+NQeERkKvAhMUdX8ZsrmFF/anArM8RaBLsD5IlKlqh81S8Km5+vv9gFVLQaKReQbYBjgr4XAlzbfADyqngPoGSKyE+gPLG+eiM2uyT+/WuOhoTQgRUR6iUgocDUwt84+c4HrvGffxwKFqrqnuYM2oeO2WUQSgA+Aa/3422Ftx22zqvZS1SRVTQLeA37kx0UAfPvd/hdwuoiEiEhbYAywqZlzNiVf2pyNpweEiHQF+gE7mjVl82ryz69W1yNQ1SoRuQP4HM+Ig5dVdYOIzPI+/iyeESTnAxlACZ5vFH7LxzY/CHQGnvF+Q65SP5650cc2tyq+tFlVN4nIZ8BaoAZ4UVXrHYboD3z8d34YeEVE1uE5bHKfqvrt9NQi8hZwBtBFRHKBh4A24Nznl00xYYwxAa41HhoyxhhzAqwQGGNMgLNCYIwxAc4KgTHGBDgrBMYYE+CsEJgWyztj6GoRWS8i73rHxfv63Jki8tQJvt+RY9z/WxE52/vzAhFJ9f48T0Q6ercfnch7HSfHY95ZNB+rc/9MEcnz/p1sFpGf+PBa32c35lhs+KhpsUTkiKq29/78BrBCVf9a6/FgVa0+xnNnAqmqekdj3q+BfRYA96pqeq37koBPVHWwr+91nPcoAmJUtbzO/TPxtklEOgNbgFNUNaeelzHGZ9YjMP7iW6CPd679r0XkTWCdiISLyD9FZJ2IrPLOpXRUvIh85p3L/qGjd4rIR94J2TaIyK2130RE/iIiK0XkSxGJ8d73itSzloGIZIpIF+BRoLf3m/pjIvK6iEyttd8bInJxneeKd9/13uzTvPfPBdoBy47eVx/vFCEZQHfv8x4UkTTv6z0v3qsGa2f35v2Nt33rRKS/D3/vJgBYITAtnoiE4FlDYZ33rtHAL1V1IHA7gKoOAaYDr4pIeK39fgAMB648ekgHuFFVR+KZi+gu77dr8HwAr1TVEcBCPFd0+uJ+YLuqDlfVn+GZz+kGb/YoPGtBzKvznMu8uYYBZwOPiUh3Vb0YKPW+1tsN/J0kAOF4riAGeEpVR3l7JRHAhcd46gFv+/4B3Otj+0wrZ4XAtGQRIrIaSMczn8xL3vuXq+pO78/jgdcBVHUzkIVnKmaAL1Q1X1VL8cyzdHTG1btEZA2edRnigRTv/TXA0Q/f2TRyhlZVXYin9xKLpzi9r6p158cfD7ylqtWqug9P4Rnlw8tPE5ENeObS+buqlnnvnyQiy7zTLJwJDDrG8z/w/rkCSPK5UaZVa3VzDZlWpVRVh9e+w3vEo7j2XQ08v+4JMBWRM/B8Ax+nqiXeY/7h1O9kTqC9jqc3cjVwYz2PN3Ze7Le95wjGAZ+KyHzgEPAMnvMHOSLya47dpqPnHaqx///Gy3oExt99g+cDFxHpCyTgOYkKcI6IdBKRCOASYBEQBRz0FoH+eJb6OyoIOHouYAbwnY8ZDgORde57BbgbQFU3HCP3NBEJ9p6LmMAJTJusqkvwFJsf838f+gdEpD3/1wZjfGLfCIy/ewZ41ntIpAqYqarl3p7Dd3g+LPsAb6pqune/WSKyFk/BWFrrtYqBQSKyAigEjnmytjZVzReRReJZbHy+qv5MVfeJyCbgo2M87UNgHJ41eBX4uaruPaGWwx+BlcAfgBfwnEPJxDN1szE+s+GjxjjAe83DOmCEqha6nceYhtihIWOamPcCrs3Ak1YEjD+wHoExxgQ46xEYY0yAs0JgjDEBzgqBMcYEOCsExhgT4KwQGGNMgPv/DDkQJBoP+9cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "gini_impurities = [1 - (v @ v) for v in vectors]\n",
    "plt.plot(prob_rain, gini_impurities)\n",
    "plt.xlabel('Probability of Rain')\n",
    "plt.ylabel('Gini Impurity')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gini Impurity is a standard measure of class imbalance. Lets compute the impurities associated with _Autumn_ and _Wetness_. This will require us to compute weighted mean of impurities associated with each split.\n",
    "\n",
    "**Listing 22. 33. Computing each feature’s Gini Impurity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "When we split on Autumn, the Impurity is 0.45.\n",
      "When we split on Wetness, the Impurity is 0.04.\n"
     ]
    }
   ],
   "source": [
    "def compute_impurity(y_a, y_b): \n",
    "    v_a = get_class_distribution(y_a)\n",
    "    v_b = get_class_distribution(y_b) \n",
    "    impurities = [1 - v @ v for v in [v_a, v_b]] \n",
    "    weights = [y.size, y_b.size]\n",
    "    return np.average(impurities, weights=weights) \n",
    "\n",
    "fall_impurity = compute_impurity(y_fall_a, y_fall_b)\n",
    "wet_impurity = compute_impurity(y_wet_a, y_wet_b)\n",
    "print(f\"When we split on Autumn, the Impurity is {fall_impurity:0.2f}.\")\n",
    "print(f\"When we split on Wetness, the Impurity is {wet_impurity:0.2f}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Impurity is minimized when we split on _Wetness_. Going forward, we will split on features whose Gini Impurity is minimized.\n",
    "\n",
    "**Listing 22. 34. Sorting features by Gini Impurity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The feature with the minimal impurity is: 'is_wet'\n"
     ]
    }
   ],
   "source": [
    "def sort_feature_indices(X, y):\n",
    "    feature_indices = range(X.shape[1])\n",
    "    impurities = []\n",
    "    \n",
    "    for i in feature_indices: \n",
    "        y_a, y_b = split(X, y, feature_col=i)[-2:]\n",
    "        impurities.append(compute_impurity(y_a, y_b))\n",
    "    \n",
    "    return sorted(feature_indices, key=lambda i: impurities[i]) \n",
    "\n",
    "indices = sort_feature_indices(X_rain, y_rain)\n",
    "top_feature = feature_names[indices[0]]\n",
    "print(f\"The feature with the minimal impurity is: '{top_feature}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `sorted_feature_indices` function will prove invaluable as we train nested if/else models with more than two features.\n",
    "\n",
    "### 22.1.3 Training If/Else Models with More Than Two Features\n",
    "\n",
    "We’ll now train a more complicated model that will predict whether or not it will rain tomorrow. \n",
    "\n",
    "The model will rely on the following three features:\n",
    "\n",
    "1. Has it rained at any point today?\n",
    "\n",
    "2. Is today a cloudy day?\n",
    "\n",
    "3. Is today an autumn day? \n",
    "\n",
    "Below, we'll simulate a training set `(X_rain, y_rain)` based on the probabilistic relationships between the features. \n",
    "\n",
    "**Listing 22. 35. Simulating a three-feature training set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)\n",
    "def simulate_weather():\n",
    "    is_fall = np.random.binomial(1, 0.25)\n",
    "    is_cloudy = np.random.binomial(1, [0.3, 0.7][is_fall])\n",
    "    rained_today = np.random.binomial(1, [0.05, 0.4][is_cloudy])\n",
    "    if rained_today:\n",
    "        rains_tomorrow = np.random.binomial(1, 0.5)\n",
    "    else:\n",
    "        rains_tomorrow = np.random.binomial(1, [0.05, 0.15][is_fall])\n",
    "    \n",
    "    features = [rained_today, is_cloudy, is_fall]\n",
    "    return features, rains_tomorrow\n",
    "\n",
    "X_rain, y_rain = [], []\n",
    "for _ in range(1000):\n",
    "    features, rains_tomorrow = simulate_weather()\n",
    "    X_rain.append(features)\n",
    "    y_rain.append(rains_tomorrow)\n",
    "    \n",
    "X_rain, y_rain = np.array(X_rain), np.array(y_rain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The columns in `X_rain` correspond to features `'is_fall'`, `'is_cloudy'`, and `'rained_today'`. We can sort these features by Gini Impurity in order to measure how well they split the data.\n",
    "\n",
    "**Listing 22. 36. Sorting three features by Gini Impurity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Features sorted by Gini Impurity:\n",
      "['is_fall', 'is_cloudy', 'rained_today']\n"
     ]
    }
   ],
   "source": [
    "feature_names = ['rained_today', 'is_cloudy', 'is_fall']\n",
    "indices = sort_feature_indices(X_rain, y_rain)\n",
    "print(f\"Features sorted by Gini Impurity:\")\n",
    "print([feature_names[i] for i in indices])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting on the _Autumn_ feature yields the lowest Gini Impurity. Meanwhile, _Cloudiness_ ranks second. Below, we'll train a two-feature model on just the _Autumn_ and _Cloudiness_ features. We'll also set the split-column to _Autumn_, since _Autumn_ has the lowest Gini Impurity.\n",
    "\n",
    "**Listing 22. 37. Training a model on the two best features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prediction = 0\n",
      "\n",
      "This statement is 74% accurate.\n"
     ]
    }
   ],
   "source": [
    "skip_index = indices[-1]\n",
    "X_subset = np.delete(X_rain, skip_index, axis=1)\n",
    "name_subset = np.delete(feature_names, skip_index)\n",
    "split_col = indices[0] if indices[0] < skip_index else indices[0] - 1\n",
    "model, accuracy = train_nested_if_else(X_subset, y_rain,\n",
    "                                       split_col=split_col,\n",
    "                                       feature_names=name_subset)\n",
    "print(model)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our trained model is frivolously simple. Its also only 74% accurate. That accuracy is not disastrous, but we can definitely do better. Ignoring the _Rainy_ feature has limited our predictive capacity. We must incorporate all three features in order to raise the accuracy score. We must incorporate all three features in order to raise the accuracy score.  We can incorporate all three features thusly:\n",
    "\n",
    "1. First, we’ll split on the feature with the lowest Gini Impurity.\n",
    "2. Next, we’ll train two nested models, using the `train_nested_if_else` function.\n",
    "3. Finally, we'll combine both _Model A_ and _Model B_ into a single coherent classifier.\n",
    "\n",
    "Lets start by splitting on the _Autumn_ feature.\n",
    "\n",
    "**Listing 22. 38. Splitting on the feature with the lowest Impurity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_a, X_b, y_a, y_b = split(X_rain, y_rain, feature_col=indices[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, lets train a nested model on `(X_a, y_a)`. This training set contains all of our non-autumn observations.\n",
    "\n",
    "**Listing 22. 39. Training a model when the season is not autumn**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If it is not autumn, then the following nested model is 88% accurate.\n",
      "\n",
      "if is_cloudy == 0:\n",
      "    prediction = 0\n",
      "else:\n",
      "    if rained_today == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n"
     ]
    }
   ],
   "source": [
    "name_subset = np.delete(feature_names, indices[0])\n",
    "split_col = sort_feature_indices(X_a, y_a)[0]\n",
    "model_a, accuracy_a = train_nested_if_else(X_a, y_a, \n",
    "                                           split_col=split_col,\n",
    "                                           feature_names=name_subset) \n",
    "print(\"If it is not autumn, then the following nested model is \"\n",
    "      f\"{100 * accuracy_a:.0f}% accurate.\\n\\n{model_a}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will train a second `model_b` based on the autumn observations stored within `(X_b, y_b)`.\n",
    "\n",
    "**Listing 22. 40. Training a model when the season is autumn**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "If it is autumn, then the following nested model is 79% accurate.\n",
      "\n",
      "if is_cloudy == 0:\n",
      "    prediction = 0\n",
      "else:\n",
      "    if rained_today == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        prediction = 1\n"
     ]
    }
   ],
   "source": [
    "split_col = sort_feature_indices(X_b, y_b)[0]\n",
    "model_b, accuracy_b = train_nested_if_else(X_b, y_b, \n",
    "                                           split_col=split_col,\n",
    "                                           feature_names=name_subset)\n",
    "print(\"If it is autumn, then the following nested model is \"\n",
    "      f\"{100 * accuracy_b:.0f}% accurate.\\n\\n{model_b}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets combine our models into a single nested statement.\n",
    "\n",
    "**Listing 22. 41. Combining the models into a nested statement**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if is_fall == 0:\n",
      "    if is_cloudy == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        if rained_today == 0:\n",
      "            prediction = 0\n",
      "        else:\n",
      "            prediction = 1\n",
      "else:\n",
      "    if is_cloudy == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        if rained_today == 0:\n",
      "            prediction = 0\n",
      "        else:\n",
      "            prediction = 1\n",
      "\n",
      "This statement is 85% accurate.\n"
     ]
    }
   ],
   "source": [
    "nested_model = combine_if_else(model_a, model_b, \n",
    "                               feature_names[indices[0]])\n",
    "print(nested_model)\n",
    "accuracies = [accuracy_a, accuracy_b]\n",
    "accuracy = np.average(accuracies, weights=[y_a.size, y_b.size])\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were able to generate a nested three-feature model. That process was very similar to how we trained a nested two-feature model. In this manner, we can extend our logic to train an arbitrary _N_ feature model.\n",
    "\n",
    "**Listing 22. 42. Training a nested model with N features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "if is_fall == 0:\n",
      "    if is_cloudy == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        if rained_today == 0:\n",
      "            prediction = 0\n",
      "        else:\n",
      "            prediction = 1\n",
      "else:\n",
      "    if is_cloudy == 0:\n",
      "        prediction = 0\n",
      "    else:\n",
      "        if rained_today == 0:\n",
      "            prediction = 0\n",
      "        else:\n",
      "            prediction = 1\n",
      "\n",
      "This statement is 85% accurate.\n"
     ]
    }
   ],
   "source": [
    "def train(X, y, feature_names):\n",
    "    if X.shape[1] == 1:\n",
    "        return train_if_else(X, y, feature_name=feature_names[0])\n",
    "    \n",
    "    indices = sort_feature_indices(X, y)\n",
    "    X_subset = np.delete(X, indices[-1], axis=1)\n",
    "    name_subset = np.delete(feature_names, indices[-1])\n",
    "    simple_model, simple_accuracy = train(X_subset, y, name_subset) \n",
    "    if simple_accuracy == 1.0:\n",
    "        return (simple_model, simple_accuracy)\n",
    "   \n",
    "    split_col = indices[0]\n",
    "    name_subset = np.delete(feature_names, split_col)\n",
    "    X_a, X_b, y_a, y_b = split(X, y, feature_col=split_col)\n",
    "    model_a, accuracy_a = train(X_a, y_a, name_subset)\n",
    "    model_b, accuracy_b = train(X_b, y_b, name_subset)\n",
    "    accuracies = [accuracy_a, accuracy_b]\n",
    "    total_accuracy = np.average(accuracies, weights=[y_a.size, y_b.size])\n",
    "    nested_model = combine_if_else(model_a, model_b, feature_names[split_col]) \n",
    "    if total_accuracy > simple_accuracy:\n",
    "        return (nested_model, total_accuracy)\n",
    "    \n",
    "    return (simple_model, simple_accuracy)\n",
    "\n",
    "\n",
    "model, accuracy = train(X_rain, y_rain, feature_names)\n",
    "print(model)\n",
    "print(f\"\\nThis statement is {100 * accuracy:.0f}% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The branching if/else statements in our trained output resemble the branches of a tree. Consequently, trained if/else conditional classifiers are referred to as **Decision tree classifiers**. Scikit-Learn includes a highly-optimized Decision tree implementation.\n",
    "\n",
    "## 22. 2 Training Decision Tree Classifiers Using Scikit-Learn\n",
    "\n",
    "In Scikit-Learn, Decision tree classification is carried out by the `DecisionTreeClassifier` class.\n",
    "\n",
    "**Listing 22. 43. Importing Scikit-Learn’s `DecisionTreeClassifier` class**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll initialize the class as `clf`. Afterwards, we'll train `clf` on the two-bulb system introduced at the beginning of the section. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier()"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualize the trained classifier using a Decision tree diagram. Scikit-Learn includes a `plot_tree` function, which leverages Matplotlib in order to carry out that visualization.\n",
    "\n",
    "**Listing 22. 45. Displaying a trained Decision tree classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADnCAYAAAC9roUQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA3eklEQVR4nO3de1iU5br48e8rYCoekqhUDMkT4sxwEFQEzROeUqHtotNWE1dqrtS1TI0OFLrYepm/1k5NO6llpWhX9VuVWi5tG5qntBQETf0FW3IHZYoLBM0UuH9/sJwNziggwwDD/bmu97pg5n2f95mbZ+55eOZ9ntcQEZRSSjlHk7qugFJKNSaadJVSyok06SqllBNp0lVKKSfSpKuUUk6kSVcppZxIk65SSjmRJl2llHIiTbpKKeVEmnSVUsqJ3Ou6Aqr+ad68+S+XL1++u67r4QqaNWt25rfffmtX1/VQ9Yehay+o6xmGIdouHMMwDETEqOt6qPpDhxeUUsqJNOkqpZQTadJVVZKUlITJZMJisRAWFsapU6cqPSY3N5fx48cDkJaWxvbt22+6/86dO4mNjbX73MGDBzGZTHTt2pWkpKTqv4BbkJSURNeuXTGZTBw8eNDm+ezsbFq0aEFwcDDBwcEkJiY6pV6qYdMv0lSl9u/fT0pKCmlpaXh4ePDTTz/h6elZ6XEdOnQgOTkZKEu6R48eZfjw4bdUhxkzZrBx40ZMJhORkZGMGzcOs9lcpWMvXLhA69atq3W+jIwMvvjiC06cOMHx48d5/PHH7Sbenj178t1331WrbNW4aU9XVernn3/G29sbDw8PADp27Ejbtm1JTk4mISEBgPj4eEaMGAHAhx9+yDPPPEN2djZhYWGUlJSQmJjI+++/T3BwMNu2bePChQuMHz+ewMBAgoKC2LdvHwAFBQU88MADdO/enTlz5gBlPebi4mICAwNxc3PjkUceYfPmzTetc15eHkuXLsVsNrNp06Zqv+bNmzfz6KOP4u7ujsVi4cqVK/z888/VLkep62nSVZUaNmwYJ06cwGKxMHv2bGvPrn///uzZsweAw4cPU1BQQElJCfv27SMyMtJ6vJubG0lJSTz22GOkpaUxYsQIkpKSuPfee0lPT+fw4cOYTCagrEe8evVqjh49yubNmzl9+jS5ubn4+PhYy/Px8SEnJ8duXb/++mvGjx9Pv379KCgoYOvWrUyYMAGAtWvXWocCym/2hiuqes6TJ08SHBxMVFQUaWlp1Yysaox0eEFVqlWrVqSmprJz506+/PJLoqKi+PDDDxk+fDi5ubkUFhZiGAZhYWGkp6ezb98+XnjhBYqKim5Y5ldffWXtgbq5udGmTRsAwsPDufPOOwEwm82cPn2a2267zeZ4w7C9CuvPf/4zGzZs4K233mLdunU0aVKxTzF58mQmT55cpdds75K568/Zvn17fvzxR7y8vNixYwexsbFkZmZWqXzVeGnSVVXi7u5OVFQUUVFReHt7s2nTJoYPH06vXr1YtWoVYWFhBAUFsX37di5fvoy3t/dNky7YT5zlE6ybmxvFxcV07ty5Qi8zJyeH9u3b2xw7Z84cWrRowfPPP89XX33FlClTCAkJsT6/du1ali9fbnPcuHHjbL4Eu75na++ct912m7W+Q4cOxcPDg3PnzuHt7X3T160aNx1eUJU6efIkWVlZQFkP8NixY3Tq1AkoG2JYtmwZkZGRREZGsnLlSvr06WNTRqtWrSgsLLT+HhUVxVtvvQVASUkJFy5cuOH5O3TogJubG+np6ZSUlPDBBx8wduxYm/38/Px46aWXOHr0KAMHDmTevHn07t3bOl48efJk0tLSbDZ7Vx2MGTOGjRs3UlxcTEZGBh4eHnTo0KHCPmfPnqWkpASA9PR0Ll26xB133HHTWCqlSVdVqqioiAkTJmAymTCbzZSWljJz5kwAIiMjycnJISIignvuuQfDMCqM514zaNAgDh8+TEhICNu2bePFF18kMzMTi8VCaGgox44du2kdVq5cyaOPPkr37t0ZOXIkFovlhvt6eHjw0EMPsWPHDjZu3FilKy2uFxgYyMiRI/H39+eRRx5hxYoVQNlY7/333w+UjR8HBgYSHBzM1KlTSU5Ottt7V6o8nQasbOg0YMfRacDqetrTVUopJ9Kkq5RSTqRJVymlnEiTrqoTgwcPdsg+lcnKyiIsLIyuXbsyffp0u9ffxsXF0blzZ+tkidzc3BqfV6kb0aSr6kRKSopD9qlMfHw8CxYsIDMzk3PnzvH555/b3e/VV1+1XkJ2/aVhSjmSJl1Vq1544QX8/f0ZNWoUQ4cOZefOnQDWCQQ7d+5k2LBhNustlN/nVokI+/fvZ/To0QBMmDCh0jUblKptOiNN1ZqDBw+yY8cOjh49yq+//oq/v7/d/dLS0vj+++9p06YNJpOJ2bNn4+vre8Ny+/bty++//27z+Pbt27nrrrusv+fl5eHl5WW9dvZmazbMmzePhIQEoqOjSUpK0uttVa3RpKtqzb59+4iOjsbDwwMfHx+7kybA/noLN0u6Bw4cqNL5q7J+AsDixYtp164dly5d4uGHH+a9994jLi6uSudQqro06apaIyIVktyNJlzYW2/hZqra0/X29ub8+fPWetxozYZrj3l6ejJx4kR2796tSVfVGh3TVbUmIiKCTZs2UVxcTG5uLvv373dIuQcOHLC7hkL5hAtlvdrw8HDrl2fJycl212y4tk5uSUkJmzdvti4zqVRt0KSrak3fvn0ZOHAgFouFWbNmERYWVu07ONTUkiVLmD9/Pl26dKFt27bWL9USExOtS0uWX0y9TZs2TJ061al1VI2Lrr2gbDhy7YWLFy/i6enJr7/+Sr9+/UhLS6NVq1YOKbsh0LUX1PV0TFfVqj/+8Y+cPHmSK1eusGjRokaVcJWyR3u6yoauMuY42tNV19MxXdVgLFiwgJUrVzr9vEeOHMHd3Z0tW7Y4/dzK9WjSVeomRITnnnuOYcOG1XVVlIvQpKtq5OLFi4waNYqgoCDMZjPbtm0DYNq0aYSGhmIymXjzzTet+3t7e/PUU0/Ro0cPxo8fz9atWwkPD8disZCdnQ2ULUDzpz/9yXq8vUvNfvjhB4YNG0ZoaCjDhw+3XvYVHx9PQEAAgYGBLFy4sMavb926dQwZMoS77767xmUpBZR9kuumW/mtrFlUzccffyxTp04VEZHS0lIpKCgQEZG8vDwREbl8+bIEBwfL2bNnRcoKl127dklpaan06tVLZsyYISIiy5Ytk/j4eBERmTRpkjzwwANSWloqhw4dEpPJJCIi8+fPlxUrVoiISFRUlGRnZ4uIyAcffCBPPPGE5OXlib+/v5SWloqISH5+vk1933nnHQkKCrLZ/vrXv9rsW1BQIAMGDJDff/9dJk2aJJs3b65yXK75Vyzr/G+qW/3Z9OoFVSMWi4W5c+eSkJBATEyM9aaUGzduZM2aNZSUlHD69Gl++OEHvL29admyJffddx8AJpOJIUOGWMsp36N96KGHMAyDXr16cfXqVfLz863PFRYWsnfvXmJiYoCySQ1+fn60bt2apk2bMm3aNGJiYhg5cqRNfatzG/b58+fzzDPP0LRp01uKjVL2aNJVNdK9e3cOHTrEli1bmDlzJo899hijR4/m9ddf55tvvqFVq1aMHDnSOm23/JTfJk2aWBNakyZNbjj9V8R2OnG7du1IS0uz2ffbb79l+/btrF+/nnfffZePP/64wvPVuQ37oUOH+OSTT5gxYwbnzp1j69atrFu3juHDh1ctOErZoUlX1Uhubi5eXl5MmjQJd3d3UlJSGDBgAJ6enrRs2ZLs7Gz27NlT7XI/+ugjHnnkEVJTU2natClt2rSxPte6dWvuvPNOvvjiC+6//36uXr1KZmYm99xzD5cuXWLs2LH06tWLgQMH2pRbnZ7u119/bf05Li6O2NhYTbiqxjTpqhrJyMhg3rx5uLm50bx5c95++2169uyJv78/ZrOZ7t27069fv2qX6+vrS2RkJPn5+bz99ts2zycnJzN9+nSee+45iouLefrpp7n99tuJiYmx9qqXLFlS49enlKPp5Ahlo64nR1zrVY4ZM6bO6uAoOjlCXU8vGVNKKSfSnq6yUdc9XVeiPV11Pe3pKqWUE2nSVUopJ9Kkq2rdu+++y7x585x6zuzsbFq0aMGAAQOAsptf9uvXD4vFQlhYGLt37660jH//938nICAAi8XCCy+8YH08MTGRdu3a1cniO6rh00vGlMvq2bOnNbl6enqyYcMG7r33Xk6cOMGYMWPIzMy86fGTJk1iw4YNFBcXM2zYMFJSUhg8eDBJSUk0aaL9FXVrtOWoaouPj2ft2rXW3ydOnMjWrVvJysqif//+hISE0K9fP06cOGFzbFxcnHWJxKKiIvz8/AAoLi7mL3/5C3369CEoKIhPP/3UoXXu1q0b9957LwD+/v4UFBRQUlJy02NGjBgBgLu7O4GBgfz0008OrZNqnDTpqmp78MEH+eijjwC4cuUKe/bsISoqivbt27Njxw5SU1N5+eWXef7556tc5po1a/Dz8+PgwYPs2rWLZ5991uaOvxkZGQQHB9ts0dHR1ar/Z599RmhoKG5ublXav7CwkM8//9zuDDelqkuHF1S19e7dm8zMTPLz89m7dy+DBg3Cw8ODixcv8uSTT5Kenk6TJk3s3ib9Rr788kuOHTvGe++9B8ClS5fIycmhc+fO1n0sFovd9RaqIysri/j4eL744osq7S8ixMXF8eSTT+Lr61ujcysFmnTVLYqOjmbTpk2kpKQQGxsLwNKlS+nSpQvJycmcOXOG8PBwm+Pc3d0pLS0FqJCURYRVq1ZZVyCzJyMjg4kTJ9o87uvra72z782cP3+eBx54gDfffJOuXbtWuj+UDaW0bduWOXPmVGl/pSqjwwvqljz44INs2LCBXbt2We+qcOHCBdq1a4dhGKxbt87ucZ06dbL2Vj/77DPr41FRUbzxxhvWcVZ7PdprPd3rt6ok3CtXrvBv//ZvzJkzx7qc5DVDhw4lJyfH5pg333yTtLQ03njjjUrLV6qqNOmqW9K3b1+OHz9O//79rcszTp8+nTfeeIOIiAgKCwvtHjdlyhS2bNlCeHi49U4RAE888QQ+Pj4EBwdjNptJSkpyaH0//PBDvvnmG5YvX24dC87Pz0dEyMzMxMvLy+aYmTNnkp2dTe/evQkODub99993aJ1U46TTgJUNV5gGnJ2dTWxsLN99991N9zt+/DirV6/mlVdeqVb5CxYswNvbm5kzZ950P50GrK6nPV3lktzc3Dhz5ox1csSNBAQEVDvhJiYmsn79ejw9PWtSRdVIaU9X2XCFnm59oT1ddT3t6Sqn8PPzo6ioqK6rwYYNGwgICKBHjx5ERESQmppqfW758uX07NmTGTNmcOLECUJCQggJCSEvL68Oa6xcjfZ0lY3a6On6+flx9OhRWrZs6dByq+PgwYNMmDCBnTt30qFDB3bv3s1jjz3GsWPHaNGiBT169GDv3r3ccccdvPTSS3h4eDB37twanVN7uup62tNVDrd69WosFgtBQUE8++yzNs+PGTOGXr16YbFYrNN9L168yKhRowgKCsJsNrNt2zZKSkqYOHEiZrMZs9lcYerxrVi2bBnPP/88HTp0AGDAgAEMHDiQ5ORk/vznP/Pf//3fDB48mDVr1rBs2TKWLl1a7dluSlVGJ0coh0pPT+fVV19lz549tGnThvPnz9vs8/777+Pl5cU///lPIiIiiImJ4R//+Af33HMPW7duRUQoLCwkLS2NM2fOcPToUQAKCgpsylq0aJF1SnJ5M2fOZMqUKRUeO378uM2HQEhICMePH+fVV19l06ZN7Nu3j5YtW/LTTz9V6eoEpapLk65yqJ07d/Lwww9b795r7/rXpUuXWic0/Pjjj/zyyy9YLBbmzp1LQkICMTEx9OnTh86dO5OVlcXs2bOJjo62mdQAkJCQQEJCQu2+KKUcSIcXlMMZxo2HMFNSUti/fz8HDhzgyJEj+Pr68vvvv9O9e3cOHTpE9+7dmTlzJitXrqRt27YcOXKEiIgIXnrpJbtr8i5atMjuIjhr1qyx2TcgIIDDhw9XeCw1NZWAgICav2ilqkpEdNOtwlbWLG5NRkaGBAYGSkFBgYiI5OXliYhIp06dpLCwUD799FOJjY0VEZEDBw6IYRhy6tQpycnJkd9++01ERNavXy+PP/64nD171lrOnj17ZOjQobdcLxGR/fv3S7du3SQnJ0dERHbv3i2+vr5SVFRUoY4iIvPnz5cVK1bU6HwiIv+KZZ3/TXWrP5sOLyiHMpvNzJo1i4iICNzd3Rk1ahSLFy+2Pj9ixAhee+01goODCQoKwmKxAGWL2cybNw83NzeaN2/O22+/TU5ODnFxcZSWluLu7s6yZctqVLfw8HBefPFFhgwZgojg5eXF3//+d53koJxKLxlTNnRyhOPoJWPqejqmq5RSTqRJVymlnEiTrlJKOZEmXaWUciK9ekFVYBhGk2bNmp0xDOPuuq6LK2jWrNmZuq6Dql+0p6usDMO4Hfh/ly9fbg/8O3AWGCEihm5V24AA4L+BhUCTy5cvpxqGEVZnf1RV7+glY8rKMIy/AH2Bo8B0YLSIZNRtrRoewzDuAjYBWUA60FNEJtVtrVR9oUlXAWCUzd09Afw/wAcYB/wkIsV1WrEG6F+x9AJWAe0AE9BVRM7VacVUvaDDC+qascC9QDcgH8gAHq/LCjVg4UA2ZR9ebsBtQM0W5lUuQ3u6CgDDMDYB/YBk4CvgaxHJr9NKNWCGYTSjLJ6DgfGUrcHQtW5rpeoDTbpKKeVEOryglFJO5BLX6TZv3vyXy5cv63WlDtKsWbMzv/32W7u6rocr0LbpWK7QNl1ieEFXxXIsXRnLcbRtOpYrtE0dXlBKKSfSpKuUUk6kSVcppZzIZZNuUlISJpMJi8VCWFgYp06dqvSY3Nxcxo8fD0BaWhrbt2+/6f47d+4kNjbW7nMzZszgrrvuIizMedPuk5KS6Nq1KyaTiYMHD9o8n52dTYsWLaw3b0xMTHRa3dT/0rbZyNtmXd2czZEb191Icd++fTJo0CC5cuWKiIj8z//8j5w/f16qY+3atTJ37tyb7pOSkiJ/+MMf7D63Z88e+e677yQ0NLRa5xUR680YqyM9PV369u0rV69elfT0dOndu7fNPqdOnapSfdCbKWrbvAFtmzXfXLKn+/PPP+Pt7Y2HhwcAHTt2pG3btiQnJ5OQkABAfHw8I0aMAODDDz/kmWeeITs7m7CwMEpKSkhMTOT9998nODiYbdu2ceHCBcaPH09gYCBBQUHs27cPgIKCAh544AG6d+/OnDlzrHWIjIzkjjvuqHKd8/LyWLp0KWazmU2bNlX7NW/evJlHH30Ud3d3LBYLV65c4eeff652Oap2advUtumSSXfYsGGcOHECi8XC7Nmz+e677wDo378/e/bsAeDw4cMUFBRQUlLCvn37iIyMtB7v5uZGUlISjz32GGlpaYwYMYKkpCTuvfde0tPTOXz4MCaTCSj7V2/16tUcPXqUzZs3c/r06WrV9euvv2b8+PH069ePgoICtm7dyoQJEwBYu3at9d+t8ltSUpJNObm5ufj4+Fh/9/HxIScnx2a/kydPEhwcTFRUFGlpadWqq6o5bZvaNl1icsT1WrVqRWpqKjt37uTLL78kKiqKDz/8kOHDh5Obm0thYSGGYRAWFkZ6ejr79u3jhRdeoKio6IZlfvXVV9ZPeTc3N9q0aQOU3db7zjvvBMpuP3769Gl8fX2rVM8///nPbNiwgbfeeot169bRpEnFz8DJkyczefLkKpVV9p9XRWWLXf2v9u3b8+OPP+Ll5cWOHTuIjY0lMzOzSuUrx9C2WaYxt02XTLoA7u7uREVFERUVhbe3N5s2bWL48OH06tWLVatWERYWRlBQENu3b+fy5ct4e3vftGGDbUMBuO2226w/u7m5UVxc9ZUQ58yZQ4sWLXj++ef56quvmDJlCiEhIdbn165dy/Lly22OGzdunM0XDdf3HnJycmjfvr1NXa/Vd+jQoXh4eHDu3Dm8vb2rXGdVc9o2G3fbdMnhhZMnT5KVlQWUfcoeO3aMTp06AWX/xi1btozIyEgiIyNZuXIlffr0sSmjVatWFBYWWn+PiorirbfeAqCkpIQLFy7UuJ5+fn689NJLHD16lIEDBzJv3jx69+5tHZObPHkyaWlpNpu9b3bHjBnDxo0bKS4uJiMjAw8PDzp06FBhn7Nnz1JSUgJAeno6ly5dqtbYnqo5bZvaNl0y6RYVFTFhwgRMJhNms5nS0lJmzpwJlH2JkJOTQ0REBPfccw+GYVQYM7tm0KBBHD58mJCQELZt28aLL75IZmYmFouF0NBQjh07dtM6TJkyhX79+pGenk7Hjh355JNPbrivh4cHDz30EDt27GDjxo14enpW+zUHBgYycuRI/P39eeSRR1ixYgVQNp52//33A2VjdIGBgQQHBzN16lSSk5Pt9pBU7dG2qW1T115QNlxhfnt9oW3TsVyhbbpkT1cppeorTbpKKeVEmnSraPDgwQ7ZpzJZWVmEhYXRtWtXpk+fbvdym7i4ODp37my9NjI3N7fG51UNm7bPBqSup8Q5YuO6qZYN2bhx42Tz5s0iIvKHP/zB+nN5kyZNsvu4o+ACUy3ry+ZKbVOk7tunK7RN7ele54UXXsDf359Ro0YxdOhQdu7cCWC9XnDnzp0MGzbM7vTKml5TKCLs37+f0aNHAzBhwgQ2b95cozKVa9H22fC57OSIW3Hw4EF27NjB0aNH+fXXX/H397e7X1paGt9//z1t2rTBZDIxe/bsm8706du3L7///rvN49u3b+euu+6y/p6Xl4eXl5f1UpkbTZcEmDdvHgkJCURHR5OUlOSyl9eo/6Xt0zVo0i1n3759REdH4+HhgY+Pj91rJKH60ysPHDhQpfOX/fdUkb3GunjxYtq1a8elS5d4+OGHee+994iLi6vSOVTDpe3TNWjSLUdEKjQie40Mqj+9sqo9CW9vb86fP2+th73pkoD1MU9PTyZOnMju3bu1UTcC2j5dg47plhMREcGmTZsoLi4mNzeX/fv3O6TcAwcO2J0yWb5BQ1mvITw8nM8//xyA5ORkxo4da1PetWXxSkpK2Lx5s3VVKeXatH26Bk265fTt25eBAwdisViYNWsWYWFhtG7d2ql1WLJkCfPnz6dLly60bdvW+qVFYmKidSWp8muntmnThqlTpzq1jqpuaPt0DToN+DoXL17E09OTX3/9lX79+pGWlkarVq0cUnZD4QpTLesLR08Dbuzt0xXapo7pXuePf/wjJ0+e5MqVKyxatKhRNWhV/2n7bPi0p6tsuEJvor7QtulYrtA2dUxXKaWcSJNuLVqwYAErV6502vm+/PJLQkNDCQwMJDIystJ1VVXjpu2zbuiYrgu58847+eKLL7j77rv5r//6L5588kl27dpV19VSCtD2eU2j6+levHiRUaNGERQUhNlsZtu2bQBMmzaN0NBQTCYTb775pnV/b29vnnrqKXr06MH48ePZunUr4eHhWCwWsrOzgbJVlf70pz9Zj7d3/eQPP/zAsGHDCA0NZfjw4dZrGePj4wkICCAwMJCFCxfW6LUFBwdz9913AxAaGspPP/1Uo/KU82n7bATqesUdR2xUYyWnjz/+WKZOnSoiIqWlpVJQUCAiInl5eSIicvnyZQkODpazZ8/Kv74BkV27dklpaan06tVLZsyYISIiy5Ytk/j4eBEpW1XpgQcekNLSUjl06JCYTCYREZk/f76sWLFCRESioqIkOztbREQ++OADeeKJJyQvL0/8/f2ltLRURETy8/Nt6vvOO+9IUFCQzfbXv/71pq9z6dKl8sQTT1Q5LuXhAis51ZetOm1TRNtnZVyhbTa64QWLxcLcuXNJSEggJibGeuO/jRs3smbNGkpKSjh9+jQ//PAD3t7etGzZkvvuuw8Ak8nEkCFDrOWU7zE89NBDGIZBr169uHr1Kvn5+dbnCgsL2bt3LzExMUDZTB0/Pz9at25N06ZNmTZtGjExMYwcOdKmvtW51fU1Bw4cYNWqVezevbtax6m6p+3T9TW6pNu9e3cOHTrEli1bmDlzJo899hijR4/m9ddf55tvvqFVq1aMHDnSOhe9/Dz2Jk2a0LRpU+vPN5rTLmI7R75du3akpaXZ7Pvtt9+yfft21q9fz7vvvsvHH39c4fnq3Ooa4NSpU0ycOJFPPvnEZe+m6sq0fbq+Rpd0c3Nz8fLyYtKkSbi7u5OSksKAAQPw9PSkZcuWZGdns2fPnmqX+9FHH/HII4+QmppK06ZNadOmjfW51q1bW79EuP/++7l69SqZmZncc889XLp0ibFjx9KrVy8GDhxoU251ehL5+fnExMTw2muv6Xz3Bkrbp+trdEk3IyODefPm4ebmRvPmzXn77bfp2bMn/v7+mM1munfvTr9+/apdrq+vL5GRkeTn5/P222/bPJ+cnMz06dN57rnnKC4u5umnn+b2228nJibG2mtZsmRJjV7bypUrOXXqFE8//TQALVq0YN++fTUqUzmXtk/XpzPSHCAuLo7Y2FjGjBlTZ3VwJFeY9VNf1HXbBNdqn67QNhvdJWNKKVWXtKerbLhCb6K+0LbpWK7QNrWnq5RSTqRJ1453332XefPmOfWc2dnZtGjRggEDBgBQVFTE0KFDadmyZZXrsnDhQnx9fW3u+pqYmEi7du2cOs9e1R5tnw1bo7t6oT7r2bOn9YJxDw8P5s+fz7Fjx8jKyqrS8SNGjODxxx/HYrFUeDwpKYkmTfTzVdWMtk/HaBSvND4+nrVr11p/nzhxIlu3biUrK4v+/fsTEhJCv379OHHihM2xcXFxbNmyBSj7dPfz8wOguLiYv/zlL/Tp04egoCA+/fRTh9b5tttu47777qN58+ZVPqZ37952bxSo6jdtn41Lo0i6Dz74IB999BEAV65cYc+ePURFRdG+fXt27NhBamoqL7/8Ms8//3yVy1yzZg1+fn4cPHiQXbt28eyzz9rcUTUjI4Pg4GCbLTo62qGvTzVs2j4bl0YxvNC7d28yMzPJz89n7969DBo0CA8PDy5evMiTTz5Jeno6TZo0sXsb6hv58ssvOXbsGO+99x4Aly5dIicnh86dO1v3sVgsdqdWKlWets/GpVEkXYDo6Gg2bdpESkoKsbGxACxdupQuXbqQnJzMmTNnCA8PtznO3d2d0tJSgAqNXkRYtWqVdbERezIyMpg4caLN476+vtY7pyoF2j4bk0YxvABl/8Jt2LCBXbt2MWzYMAAuXLhAu3btMAyDdevW2T2uU6dO1t7AZ599Zn08KiqKN954g5KSEgC7PYZrPYnrt5o26KFDh5KTk1OjMlT9ou2z8Wg0Sbdv374cP36c/v37W1dimj59Om+88QYREREUFhbaPW7KlCls2bKF8PBw66LQAE888QQ+Pj4EBwdjNptJSkpyeJ1NJhNz5szhrbfeomPHjvzyyy+ICJmZmXh5ednsv2DBAjp27Mg///lPOnbsyKuvvurwOqnaoe2zEanrBX0dsVHNhaLro1OnTkloaGil+33//ffy1FNPVbv88gtWVwYXWCi6vmyu0DZF6k/7dIW22Wh6uvWdm5sbZ86csV58fiMBAQG88sor1So7MTGR9evX4+npWZMqqkZM26fj6NoLyoYrzG+vL7RtOpYrtE3t6d6An58fRUVFdV0NNmzYQEBAAD169CAiIoLU1FTrc8uXL6dnz57MmDGDEydOEBISQkhICHl5eXVYY+UM2j4bsLoe33DERi2Mm3Xq1EkKCwsdXm51HDhwQLp16yY5OTkiIvL111+Ln5+fXLx4UURE/P395dy5cyIisnjxYvnb3/7mkPPiAuNm9WWrjbYp0njbpyu0zTqvgENeRA0b9qpVq8RsNktgYKA888wzIlKxUY8ePVpCQkLEbDbLJ598IiIiRUVFMnLkSAkMDBSTyST/+Mc/pLi4WCZMmCAmk0lMJpO88847NarXo48+KmvXrq3w2KRJk2TVqlUya9Ys8fDwEIvFIqtXr5a7775bfHx8ZOzYsTU6p4hrNOz6sjki6Wr7/F+u0DbrvAIOeRE1aNhHjhwRs9lsvb30tVtdl2/U1x47f/689OjRQ0pLS+3eKvu7776TYcOGWcu2d8vqhQsX2r1l9erVq232DQ4OliNHjlR4bNmyZdZvh8vXsTpXJ1TGFRp2fdlqmnS1fVbkCm2z0cxIu5GdO3fy8MMPW2/UZ+/6wqVLl1ovGP/xxx/55Zdf7N4qu3PnzmRlZTF79myio6Ott8MuLyEhgYSEhNp9UcplaPt0PfpFGlS4HfX1UlJS2L9/PwcOHODIkSP4+vry+++/W2+V3b17d2bOnMnKlStp27YtR44cISIigpdeesnuOqOLFi2yu8jImjVrbPYNCAjg8OHDFR5LTU0lICCg5i9aNRjaPl1MXXe1HbFRg3/hMjIyJDAwUAoKCkTE9t+3Tz/9VGJjY0Wk7IsDwzDk1KlTkpOTI7/99puIiKxfv14ef/xxOXv2rLWcPXv2yNChQ2+5XiIi+/fvr/BFxe7du8XX11eKiooq1FFEhxfq61aTtimi7fN6rtA2G/3wgtlsZtasWURERODu7s6oUaNYvHix9fkRI0bw2muvERwcTFBQkHUBZnu3ys7JySEuLo7S0lLc3d1ZtmxZjeoWHh7Oiy++yJAhQxARvLy8+Pvf/95oLiJX2j5dkU6OUDZc4QL0+kLbpmO5QtvUMV2llHIiTbpKKeVEmnSVUsqJNOkqpZQTucTVC82aNTtjGMbddV0PV9GsWbMzdV0HV6Ft07FcoW26xNULtcEwjNuBE8AoEUmtZHenMgzjj8AUIFK/Gm+cDMNYCSAiM+u6LuUZhtEWOA6MFJG0Oq5OvaRJ9wYMw3gFaCki0+q6LtczDKMJcBB4RUQ21HV9lHMZhmEGvgICRKTerZNoGMZ04BFgsHYKbGnStcMwDH9gD2ASkV/ruj72GIYRCXwA9BCRi3VdH+UcRtmc4C+Bz0RkRV3Xxx7DMNyAw8B/iMjHdV2f+ka/SLPvFWBxfU24ACKyF9gNxNd1XZRTRQPtgTfruiI3IiIlwGzgb4ZhNK/j6tQ72tO9jmEY9wNLAYuIXKnr+tyMYRj3AKlAqIj8WNf1UbXLMIzbgO+B6SLyZV3XpzKGYXwMpInIwrquS32iPV3AMIw7DMNIMgyjKWW93Kfqe8IFEJH/AV4F/o9hGE0Mw6iX/26qmjEM43nDMHwo6z0ebQgJ91+eBp4yDMPHMIzJhmGE1XWF6gPt6QKGYYQDyykbIx0uIqPquEpVZhhGC8q+LZ4IpADNRORq3dZKOZJhGEeAuZS1z3ARyazjKlWZYRgLgU5AEZAhIq/XcZXqnPZ0y7QBLgHPAysMw/i//7r0pV4zDKMvZT3d/wCWAReA1nVZJ1Ur2gDTgPeASYZhTKjj+lSJYRjrgH3AYMCTstfR6GnSLdOGsk/jE5Q17B1Afl1WqIqOUFbP/wCaA8Vow3ZFdwBRwGjABGyv2+pUWTJlX/idpKz+t9dpbeoJTbplegL3Upa0eovI6w3h+kIRuSwi84B/A5oB3pR9s61cxL+uyW4JuAGJwB/q81U15YnIPwAL8N+UtcvedVuj+sElpgE7wLfAEuC5hpBsryci3xiGEQCsBv6nruujHEqA/ws82VCSbXkiUgBMNQxjH3BnXdenPtAv0pRSyol0eEEppZyo2sMLzZs3/+Xy5cu6alIVNGvW7Mxvv/3W7kbPayyrrrJYgsazOjSejlWVeF5T7eEFvedT1VV2PyeNZdVV5d5YGs+q03g6VnXu3abDC0op5USadJVSyok06SqllBPVedIdPHiwQ/apTFZWFmFhYXTt2pXp06djb6zq3LlzDB48mG7dujFu3DguX75c4/M6m8bTsTSejqOx/BcRqdZWdkjDM27cONm8ebOIiPzhD3+w/lzenDlzZMWKFSIiMnfuXOvPt+pfsXK5WIo4P56VxVI0ntXiyvGsj+/18pvTkm5CQoJ0795dRo4cKUOGDJGUlBQREbnjjjtERCQlJUWioqIkJiZGunXrJk899ZT12Gv73KrS0lJp3769lJaWiojIJ598ItOmTbPZr1u3bpKfny8iIqmpqTJ8+PAanbc2k25ji2dtJwmNp77Xa6I6Sdcp04APHjzIjh07OHr0KL/++iv+/v5290tLS+P777+nTZs2mEwmZs+eja+v7w3L7du3L7///rvN49u3b+euu+6y/p6Xl4eXlxdldzoBHx8fcnJybI4rKCigTZs2N92nPtB4OpbG03E0lpVzStLdt28f0dHReHh44OPjQ2RkpN39wsPDufPOsunZZrOZ06dP3/QPceDAgSqdX+yM6Vz7o9zsMXv71AcaT8fSeDqOxrJyTkm6IlLhRdkLDMBtt91m/dnNzY3i4uKbllvVTz9vb2/Onz9vrUdOTg7t29suxtW6dWvrJ+CN9qkPNJ6OpfF0HI1l5Zxy9UJERASbNm2iuLiY3Nxc9u/f75ByDxw4QFpams1W/o8AZZ9i4eHhfP755wAkJyczduxYm/LGjBnDunXrAFi/fr3dfeoDjadjaTwdR2NZOack3b59+zJw4EAsFguzZs0iLCyM1q2de4ODJUuWMH/+fLp06ULbtm0ZPXo0AImJiWzatAmA5557jo8++ohu3bqRmZnJlClTnFrHqtJ4OpbG03E0lpVz2toLFy9exNPTk19//ZV+/fqRlpZGq1atql1OQ1Kbay80tnjW9loBGk+7++h7vYqqs/aC0xYx/+Mf/8jJkye5cuUKixYtcvk/Qm3TeDqWxtNxNJY3p6uM1SJdZcxxdFUsx9J4OpauMqaUUvVUg026CxYsYOXKlU47X3Z2NpGRkTRr1syp53UGjaVjaTwdy9XiqTemrKLWrVvzyiuvWL/9VLdOY+lYGk/Hqu14OrSne/HiRUaNGkVQUBBms5lt27YBMG3aNEJDQzGZTLz55pvW/b29vXnqqafo0aMH48ePZ+vWrYSHh2OxWMjOzgYgLi6OP/3pT9bj7V3398MPPzBs2DBCQ0MZPnw4P//8MwDx8fEEBAQQGBjIwoULa/TavLy86Nu3Lx4eHjUqp6o0lo6l8XQsjWcNVHWRhmsbN1kE4+OPP5apU6eKSNnCEwUFBSIikpeXJyIily9fluDgYDl79qx1kYhdu3ZJaWmp9OrVS2bMmCEiIsuWLZP4+HgREZk0aZI88MADUlpaKocOHRKTySQiIvPnz7euDBQVFSXZ2dkiIvLBBx/IE088IXl5eeLv729d+OLa4hblvfPOOxIUFGSz/fWvf73hayx/3spQgwVvNJYVVRZL0XhqPOt5PK9tDh1esFgszJ07l4SEBGJiYujTpw8AGzduZM2aNZSUlHD69Gl++OEHvL29admyJffddx8AJpOJIUOGWMsp/yn30EMPYRgGvXr14urVq+Tn51ufKywsZO/evcTExABQUlKCn58frVu3pmnTpkybNo2YmBhGjhxpU9/JkyczefJkR4bAYTSWjqXxdCyN561zaNLt3r07hw4dYsuWLcycOZPHHnuM0aNH8/rrr/PNN9/QqlUrRo4caZ1DXX7+dZMmTWjatKn15xvNxRY7c7vbtWtHWlqazb7ffvst27dvZ/369bz77rt8/PHHFZ5fu3Yty5cvtzlu3LhxJCYmVvv1O5LG0rE0no6l8bx1Dk26ubm5eHl5MWnSJNzd3UlJSWHAgAF4enrSsmVLsrOz2bNnT7XL/eijj3jkkUdITU2ladOm1iXZoGzQ+8477+SLL77g/vvv5+rVq2RmZnLPPfdw6dIlxo4dS69evRg4cKBNufXp0+96GkvH0ng6lsbz1jk06WZkZDBv3jzc3Nxo3rw5b7/9Nj179sTf3x+z2Uz37t3p169ftcv19fUlMjKS/Px83n77bZvnk5OTmT59Os899xzFxcU8/fTT3H777cTExFg/aZcsWVKj13bhwgV69uzJhQsXcHNz429/+5v1C4DaoLF0LI2nY2k8b129n5EWFxdHbGwsY8aMcdo5HaW+zUhz5Vj+ax+NZxVpPB1LZ6QppVQ9Ve97ug1ZfevpNmT1sWfWkGk8HUt7ukopVU85Nem+++67zJs3z5mnJDs7mxYtWjBgwAAAioqKGDp0KC1btqxyXbKysggLC6Nr165Mnz7deguSxMRE2rVrV2fz3etDPAHWrFlDt27d8Pf3Z8uWLZWWsXDhQnx9ffH29q7wuMazjMbz1jWE93ujWHuhZ8+e7N69GwAPDw/mz5/PsWPHyMrKqtLx8fHxLFiwgDFjxhAbG8vnn3/OmDFjSEpKokmTxvfPQvl45uXl8fLLL3P48GEKCwsZNGgQI0eOxN39xk1rxIgRPP7441gslgqPazw1no5Q39/vNSohPj6etWvXWn+fOHEiW7duJSsri/79+xMSEkK/fv04ceKEzbFxcXHWT/GioiL8/PwAKC4u5i9/+Qt9+vQhKCiITz/9tCZVtHHbbbdx33330bx58yrtLyLs37/fesuPCRMmsHnzZofW6ZqGGM9t27Zx//3306pVKzp06EDPnj359ttvb3pM7969nXIjQI2nYzXEeNbH93uNku6DDz7IRx99BMCVK1fYs2cPUVFRtG/fnh07dpCamsrLL7/M888/X+Uy16xZg5+fHwcPHmTXrl08++yzNncBzcjIIDg42GaLjo6uycuxKy8vDy8vL+vMGB8fH3Jychx+HmiY8czNzcXHx8f6e23Gp7o0no7VEONZXc54v9doeKF3795kZmaSn5/P3r17GTRoEB4eHly8eJEnn3yS9PR0mjRpYvfWyTfy5ZdfcuzYMd577z0ALl26RE5ODp07d7buY7FY7E4FrA32vr0tPzXRkRpiPK+fqgm1F5/q0ng6VkOMZ3U54/1e4zHd6OhoNm3aREpKCrGxsQAsXbqULl26kJyczJkzZwgPD7c9sbs7paWlABX+SCLCqlWrrItj2JORkcHEiRNtHvf19XX4Gpje3t6cP3/e+mbIycmp1X/lGlo8fXx8Kvz7W9vxqS6Np2M1tHhWlzPe7zUeFX7wwQfZsGEDu3btYtiwYUDZNLp27dphGIb13vLX69Spk/XT67PPPrM+HhUVxRtvvEFJSQmA3U+4a5981281/QMMHTrU5l8JwzAIDw/n888/B8qmIY4dO7ZG57mZhhbP4cOH88UXX1BYWEhubi7Hjh2zrjhlL57OpvF0rIYWz5upq/d7jZNu3759OX78OP3797euHDR9+nTeeOMNIiIiKCwstHvclClT2LJlC+Hh4RXmNT/xxBP4+PgQHByM2WwmKSmpplW0YTKZmDNnDm+99RYdO3bkl19+QUTIzMzEy8vLZv8lS5Ywf/58unTpQtu2ba2D7LWhocXT29ubuXPnEhISwqBBg/jP//xP3N3dbxrPBQsW0LFjR/75z3/SsWNHXn31VYfWqTyNp2M1tHhCPXy/V3Xh3WsbN1nYuD46deqUhIaGVrrf999/L0899VS1y7/ZQsfUYBHz+qqu4llZLEXjaZfG0766iOe1zeUv4nNzc+PMmTMVLj63JyAggFdeeaVaZScmJrJ+/Xo8PT1rUsUGRePpWBpPx2oI8dS1F2qRrr3gOLpWgGNpPB2r3q+94OfnR1FRUV2cuoINGzYQEBBAjx49iIiIIDU11frc8uXL6dmzJzNmzODEiROEhIQQEhJCXl5eHdbYvoYQz4ZCY+lYDSGeTn+vV3UcQhw4ztOpUycpLCyscTk1ceDAAenWrZvk5OSIiMjXX38tfn5+cvHiRRER8ff3l3PnzomIyOLFi+Vvf/tbtc+Bk8Z0G0I8a6qyWEojapuOoPF07nu9/FbrSXfVqlViNpslMDBQnnnmGRGp+IcYPXq0hISEiNlslk8++URERIqKimTkyJESGBgoJpNJ/vGPf0hxcbFMmDBBTCaTmEwmeeedd6odmPIeffRRWbt2bYXHJk2aJKtWrZJZs2aJh4eHWCwWWb16tdx9993i4+MjY8eOrdY5aiPpNsR4iojccccdMnfuXDGbzTJkyBApKiqqVvm1kSQaayxFNJ7Ofq+X32o16R45ckTMZrP1lsjXbs9c/g9x7bHz589Ljx49pLS01O7tnb/77jsZNmyYtWx7t1leuHCh3dssr1692mbf4OBgOXLkSIXHli1bZv1Gs3wdq3Mr5vIcnXQbcjwB+eqrr0REZOLEifL+++9X67U7Okk05lheK0M0njZ1rK33evmtVlcZ27lzJw8//LD15nL2rolbunSp9SLnH3/8kV9++cXu7Z07d+5MVlYWs2fPJjo62noL5/ISEhJISEiozZdUpxpyPFu2bMngwYMBCA0N5ccff3RIubdKY+lYDTmezlbrX6TdbN5ySkoK+/fv58CBAxw5cgRfX19+//136+2du3fvzsyZM1m5ciVt27blyJEjRERE8NJLL9ldG3PRokV2F8ZYs2aNzb4BAQEcPny4wmOpqakEBATU/EXXooYaz/K34HZzc7vhbbedSWPpWA01nk5X1S6x3MK/HBkZGRIYGCgFBQUiYvsvx6effiqxsbEiUjbYbRiGnDp1SnJycuS3334TEZH169fL448/LmfPnrWWs2fPHhk6dGi1/wUob//+/RUG13fv3i2+vr7W8bH6OLzQkON5xx13WPddsWKFzJ8/v1rlVxZLaURts6axFNF4Ovu9Xn6r1eEFs9nMrFmziIiIwN3dnVGjRrF48WLr8yNGjOC1114jODiYoKAg6yLM9m7vnJOTQ1xcHKWlpbi7u7Ns2bIa1S08PJwXX3yRIUOGICJ4eXnx97//vV5fSK7xdByNpWNpPKtOJ0fUIp0c4Th6Mb9jaTwdq95PjlBKqcZKk65SSjmRJl2llHIiTbpKKeVE1b56oVmzZmcMw7i7Nirjapo1a3amsuc1llVTWSyv7aPxrBqNp2NVJZ7XVPvqBaWUUrdOhxeUUsqJNOkqpZQTadJVSikn0qSrlFJOpElXKaWcSJOuUko5kSZdpZRyIk26SinlRJp0lVLKiTTpKqWUE2nSVUopJ9Kkq5RSTqRJVymlnEiTrlJKOZEmXaWUcqL/D7dFAUY3Z/7nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.tree import plot_tree\n",
    "feature_names = ['Switch0', 'Switch1']\n",
    "class_names = ['Off', 'On']\n",
    "plot_tree(clf, feature_names=feature_names, class_names=class_names)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn provides an alternative visualization function. The `export_text` function allows us to display the tree using a simpler, more compact, text-based diagram.\n",
    "\n",
    "**Listing 22. 46. Displaying a Decision tree classifier as a string**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|--- Switch0 <= 0.50\n",
      "|   |--- Switch1 <= 0.50\n",
      "|   |   |--- class: 0\n",
      "|   |--- Switch1 >  0.50\n",
      "|   |   |--- class: 1\n",
      "|--- Switch0 >  0.50\n",
      "|   |--- Switch1 <= 0.50\n",
      "|   |   |--- class: 1\n",
      "|   |--- Switch1 >  0.50\n",
      "|   |   |--- class: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import export_text\n",
    "text_tree = export_text(clf, feature_names=feature_names)\n",
    "print(text_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus far, all of our features have been Booleans. However, in most real-world problems, the features are numeric. Fortunately, any numerical feature can be transformed into a Boolean feature. We simply need to run `feature >= thresh`, where `thresh` is some numeric threshold. In Scikit-Learn, Decision trees scan for this threshold automatically. That is why they split on inequalities.\n",
    "\n",
    "How should we select the optimal threshold for splitting a numeric feature? Well, we simply choose the threshold that minimizes the Gini Impurity.\n",
    "\n",
    "**Listing 22. 47. Choosing a threshold by minimizing the Gini Impurity**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Impurity is minimized at a threshold of 0.70\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEHCAYAAAC5u6FsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4jklEQVR4nO3deXhU5fXA8e/JRjYgIQlrwr6JsofVDQUXcEGrFrXi0gWppS1W26pVW6u21p9aq1WQWot1hboBFjdQRAWUxYAQQMIihAAJBAIkIev5/TEDHUIIM8nc3ExyPs8zT2buve+95zWYM/e97yKqijHGGOOvMLcDMMYYE1oscRhjjAmIJQ5jjDEBscRhjDEmIJY4jDHGBMQShzHGmIBEOHlyEbkY+BsQDjyvqo9U2d8b+BcwCPidqj7m3d4LmOVzaFfgflV9UkT+APwEyPPuu0dV59cUR3Jysnbu3LnuFTLG1MnGjRsB6NWrl8uRGH+sXLlyr6qmVN3uWOIQkXDgGeACIBtYLiJzVTXT57B84BfAFb5lVXUjMMDnPDuBt30O+evRJOOPzp07s2LFilrUwhgTTKNGjQJg0aJFrsZh/CMi31W33cmmqqFAlqpuUdVS4HVgvO8BqpqrqsuBshrOMxrYrKrVVsAYY0z9crKpqgOww+dzNjCsFue5FnityrYpInIjsAK4Q1X31y5EY0x9uvfee90OwQSBk3ccUs22gOY3EZEo4HLgPz6bpwHd8DRl7QIeP0nZSSKyQkRW5OXlVXeIMaaejRkzhjFjxrgdhqkjJxNHNpDm8zkVyAnwHGOBVaq65+gGVd2jqhWqWgn8A0+T2AlUdYaqpqtqekrKCc92jDEuyMjIICMjw+0wTB052VS1HOghIl3wPNy+Frg+wHNcR5VmKhFpp6q7vB+vBNbWNVBjTP2YOnUqYA/HQ51jiUNVy0VkCvABnu64L6jqOhGZ7N0/XUTa4nlO0QKoFJGpQB9VPSgisXh6ZN1a5dSPisgAPM1e26rZb4wxxkGOjuPwjq+YX2XbdJ/3u/E0YVVXtghIqmb7xCCHaYwxJgCOJo5Qt3D9HlbvOEBcswj6pSbQN7UlsZHhhIVV99zfGGOaBkscNfj02zz+vfT44SNJcVFc1r8991/axxKIMaZJkqawAmB6errWZeT4lrzDfLOzgK17C/ls015WfrefDgkxjD6tNf1TExjYMYEuyXGIWCIxpiZLliwBYOTIkS5HYvwhIitVNf2E7ZY4AlNaXsn9c9ay8rv9ZOUd5uh/vq7JcUwe1Y1ebZrTL7WlJRFjTMizxOHAXFXFpRWsyT7Asi35vPbVdnYfPAJAv9SWPHP9INJaxQb9msaEMrvjCC2WOBye5LC4tILMXQf5KHMP0z/dDMCFfdow/YbB9izEGC+b5DC0nCxx2MPxIImJCmdwp0QGd0pk7Blt+fsnWXyYuYdRjy3iwj5tOLtnCiO6JnG0BSsiTKw5yxgTkixxOKB/WgLP3TCYJxdu4p2vd/L851t5/vOtxx3TISGGyed25QfDOtkdiTEmpFjicEhYmPCrC3ryqwt6svdwCR9l7mHf4RIAikormLcmh/vmrOO/3+ziuRvSaRkb6XLExhjjH0sc9SA5vhnXDe143LZfX9SLSS+t5KPMPVw1fQkD0xK4pF87RvVq7VKUxphQt2H3QTbuPnTctuFdk2jTIjqo17GH4y771xdbef2rHWzc4/lln96+BZPO6cr4AR1cjsyY4Ds6M+6AAQNcjaOxOufRT9ieX3Tctpm3DKn1F1LrVdVAE8dROQeKuX/OOtbuLGD3wSNc2KcNPds058zuyYzodsKUXcYYcxxVpdd97/O9gR34yTldj21v1zKa2KjaNS5Zr6oGrn1CDM/flM7OA8X87u1vWLwpjw8z9/D3T7K495LT+PHZXU99EmMauAULFgDYYk4OKCytoLS8ki7JcXRLiXf0WpY4GpgOCTHMvMWzNlVBcRm3z8rgof+uZ/2uQzx2TT/rwmtC2kMPPQRY4nDC8q35ALSKi3L8Wk6uAGjqqGVMJNNvGMz3BnbgzVXZ9LrvfcY88SmZOQfdDs0Y08Cs+M6TOAakJTh+LbvjaOCiIsJ45Kp+tG0ZTUFxGQvX5/K9aV+QEBNFjzbx/P6yPnRv3dztMI0xLssvLCU5vhk92jj/98ASRwiIigjjNxf3BuBHZx3mhS+2UlRawZyMHMY8sZirB6fyl6v6EW4DCY1psvYdLqVVXP2MB7PEEWK6psTz0BV9AZh0Tld+8drXvLEym4Xr9/DwlX0Z17edyxEaY5z2w5nLWb/r+CbrfYdLGdQpoV6ub4kjhPVu24K3bjuTV5Z9x5/f28CD72YSHiac2zOF6Mhwt8Mz5gTPPfec2yGEvJLyCj7ekEu/1Jb0bnt8s1R9fXG0xBHi4ptFcOu53RjRLYkJzy3j1pdWAjCyWxI9WsczsnsyZ3ZPJr6Z/aqN+3r16uV2CCFvf2EZABOGpPGDYZ1cicHRXlUicrGIbBSRLBG5q5r9vUVkqYiUiMidVfZtE5FvRCRDRFb4bG8lIh+JyCbvz0Qn6xAq+qUmsOSu83ni+/0Z17ctm3IP8+LS77j1pZWc/9giFq7fw8EjZW6HaZq4efPmMW/ePLfDCEmHjpSxI7+I9bs9TVRJ9dDt9mQcGzkuIuHAt8AFQDawHLhOVTN9jmkNdAKuAPar6mM++7YB6aq6t8p5HwXyVfURbzJKVNXf1hRLKIwcd8LewyX83juR4lG/vqgXPzuvu4tRmabM1uOondLySoY8vICC4v99+XvzpyMZ3MnZ781ujBwfCmSp6hZvAK8D44FjiUNVc4FcEbkkgPOOB0Z5378ILAJqTBxNVXJ8M575wSAeKizlw8zd3D9nHf/3wUZmLd/BT0d14/L+7YmzJixjGrx1OQUUFJdx44hO9O3QkvhmEQysh/EaJ+PkX40OwA6fz9nAsADKK/ChiCjwnKrO8G5vo6q7AFR1l/euxdQgMS6KCUM6ck7PFGZ+sY1Xv9rO3W99w33vrGVUrxT+dGVfWgd59kxjTPD847MtANwwvBM962Gcxqk4mTiqG1QQSLvYmaqa400MH4nIBlVd7PfFRSYBkwA6dux4iqObhnYtY7h73GncfkFP5mTs5K1VO1mwPpdDR77miQkD6JAQ43aIxpgqPt+0lw27DtE5KbZBJA1w9uF4NpDm8zkVyPG3sKrmeH/mAm/jafoC2CMi7QC8P3NPUn6GqqaranpKSkotwm+8oiPDmTCkI7NuHcEPz+zCl1vz+dWsDL7cso8jZRVuh2eM8crKPcwN//ySLXsLG9QYLSfvOJYDPUSkC7ATuBa43p+CIhIHhKnqIe/7C4E/enfPBW4CHvH+nBPswJuS+y/rQ0S4MGPxFibMWEZK82bMvnUEXZLj3A7NNEIvvfSS2yGElLmrPd+1n78xnVG9Gs4XYMcSh6qWi8gU4AMgHHhBVdeJyGTv/uki0hZYAbQAKkVkKtAHSAbe9s4EGwG8qqrve0/9CDBbRH4EbAeucaoOTcVdF/fmotPbsHbnQR76byYX/XUxU87vzoQhabRu3sxm5DVBk5aWduqDzDELMvcAcFaPZCLCG86ctLaQkznO19v385N/r2Dv4VIABnZM4O/XD7LnHyYoZs2aBcCECRNcjqThKq+o5LnFWzh0pJyZS7Zy9eDUY9MM1TdbAdASh99UlYXrc/l4Yy6vfrkd8CQQX+1aRvPwFX1JdHEQkgk9No7j1JZt2ce1M5YRGS5Ehofx9HUDGX1aG1disRUAjd9EhDF92jCmTxsu7duOF77YRkn5/x6aHywuY/43u8neX8xbPx3ZoG6hjQllO/KLmPLqKgAW/+Y82rVsmHf6ljhMjUZ2T2Zk9+QTts9avp3fvvkN/R/4kDsu7MUPz+riQnTGNC7vfL2TvYdLGdalFW0b8Ngq+6poamXCkI786cq+FJZW8Md3M5m1fLvbIRkT0nYVFPP4R9/SISGGWbeOaNCdUixxmFq7flhHlv9uDM0iwvjtm99w2dOf88mGXDbnHT72Ki2vdDtMY0LCgvWeIWnj+rZ1OZJTs6YqUycpzZux7O7R/OX9Dby+fAe3zFx+3P5WcVHceWGvY+sgt20ZTSt7oN5kvfHGG26H0GBtzj1MXFQ494w7ze1QTskSh6mzxLgoHrmqH7df0JNlW/Yd2745r5B/LN7CPW9/c9zxl/Zrx68u6EnXlPj6DtW4LDn5xOdlxmN3wRHaJ8Q06CaqoyxxmKBp0yKa8QM6HLftp+d24/OsvVRUKgeKSnn76528u2YXSzbvY+W9Y0LifxITPDNnzgTg5ptvdjWOhii/qDRk7sbtGYdxVExUOBf0acPFZ7Tl2qGe+bF+cX538gtLmTorg7IKewbSlMycOfNY8jDHyy8MncRhdxym3v18dA++2VnAnIwcFn+bx5DOrQDPMrh3XtSL9jZK3TQhn2zM5a4315B7qIShXVq5HY5fLHGYehcZHsaMG9OZsXgL87/Zxfb8IgC27i1kbU4B1wxO49qhaTSPjnQ5UmOct2hDLgXFZVw/tCPXDwuNJSBsyhHTYLzz9U5+++YaSsorad4sgrd/NpLurRvG+gMmOGzKkeM9/uFGnv44i/6pLZkz5Sy3wznByaYcsWccpsG4YmAH1j5wEU9fN5AKVa6ZvpSCorJTFzQmRH2y0TN24+4Q6ILryxKHaVAiw8O4rH97nr5uIPuLyrhq+hK3QzJBNH/+fObPn+92GA2CqrIlr5BbzuzM8K5JbocTEEscpkEafVobrh6cSlbuYXZ4n4GY0BcbG0tsbKzbYTQIRaUVFJVWNOg5qU7GEodpsCaf2w2AB+atczkSEyzPPvsszz77rNthNAj5hZ41b0JxaQJLHKbB6pYSR+ekWL7I2nfctO4mdM2ePZvZs2e7HYbrVJVt+woBaBVricOYoBER7hl3GsVlFcz8Ypvb4RgTNDOXbGPiP78CILl5M5ejCZwlDtOgjeyeTFREGH9+bwNLsva6HY4xdVZQVMaSzftIjm/GU9cNpF+Hlm6HFDAbAGgatPhmEXx0+zlc+NfF/OzVVcetiPbjs7vwvUGpLkZnTGCOlFVw1l8+5lBJOaN7t+by/u3dDqlWHL3jEJGLRWSjiGSJyF3V7O8tIktFpERE7vTZniYin4jIehFZJyK/9Nn3BxHZKSIZ3tc4J+tg3NcpKY7nb0onvXMr2ifE0D4hhsLSch6Yl8kds1dzx+zV3Pmf1azeccDtUI05qd0FR/h8014OlZRzy5mdeWD86W6HVGuO3XGISDjwDHABkA0sF5G5qprpc1g+8AvgiirFy4E7VHWViDQHVorIRz5l/6qqjzkVu2l4zu6Rwtk9Uo59zsw5yJRXVx2bxj3vUAlFpeU8+4PBboVo/NBUR4zvyC/inP/7hKMTdVw3tCOpiaHbLdnJpqqhQJaqbgEQkdeB8cCxxKGquUCuiFziW1BVdwG7vO8Pich6oINvWdO09Wnfgo/vHHXs891vrWFORg77DpeQFB96DxtN47Z8Wz6qcPfY3vRp34KebUJ7Kh0nm6o6ADt8Pmd7twVERDoDA4EvfTZPEZE1IvKCiCTWKUrTKPzorK4UlVYwa8WOUx9sXPPYY4/x2GNNr7FgxuItgGe5Zd8751DlZOKoboWegGZUFJF44E1gqqoe9G6eBnQDBuC5K3n8JGUnicgKEVmRl5cXyGVNCOreOp5ebZrz6PsbKSotdzsccxLvvvsu7777rtth1LuS8kp6t23eaGZ8djJxZANpPp9TgRx/C4tIJJ6k8YqqvnV0u6ruUdUKVa0E/oGnSewEqjpDVdNVNT0lJfQzvDm18QM9PVQm/vMr/vrRty5HY8z/7C8qPbbuTGPgZOJYDvQQkS4iEgVcC8z1p6B41hP9J7BeVZ+osq+dz8crgbVBiteEuB+e2YXRvVuzu+AIT3+8iRc+38p+77QOxrilolIpKC4jMbZx3G2Ag4lDVcuBKcAHwHpgtqquE5HJIjIZQETaikg28CvgXhHJFpEWwJnAROD8arrdPioi34jIGuA84Han6mBCS3RkOP+8eQhPXz8QgD++m8l9c9bSFNacMQ1XfmEpqoTMsrD+cHQAoKrOB+ZX2Tbd5/1uPE1YVX1O9c9IUNWJwYzRND6DOiay7oGLuW/OWt5Ymc0VAzowpk8bt8MyQExM418WuLi0go17Dh37/O1uz/tOyXFuhRR0NnLcNEoxUeH8btxpvLEym9++uYblvccQFlbtdxFTj9577z23Q3DcH99dx2tfndi7L9S74PqyxGEarcS4KG4Y3pGXl23nvjlruaRfO0Z2S3Y7LNPIZe8vpltKHPde0ufYtlZxUXRIaDx3W5Y4TKP2+8tO56ut+bzy5XZe+XI7Z3RowT1jT2Nkd0sgbnjwwQcBuO+++1yOxDn7i0rp2CqW83q3djsUx9jsuKZRiwwP48Pbz+XD28/h4tPbsnbnQa5//ktWbMt3O7QmaeHChSxcuNDtMBy1v7CMxBBcYyMQljhMk9CzTXOmTxzMG5NHADDl1a+5dsZSNvk8xDSmrlSV/MLSkFzVLxCWOEyTkt65FU98vz9dU+JYtiWf5z/b6nZIphFQVbJyD7F0yz6KyyronBS6Exj6w55xmCbne4NS+d6gVH784gq+siYrEwRzV+fwy9czjn3u0Yh6UFXH7jhMk3V2j2S27i3k+88ttUGC9SQpKYmkpCS3wwi6zJyDRIWHMe0Hg/j3D4cyrEvjmV6kOnbHYZqs76en8f7a3Szdso8/zF3H5QM6MLiTTbbspDfffNPtEByxOa+QzsmxjO3b7tQHNwJ2x2GarJiocP55czodW8Xy4tLvuGraEiY8t5Q9B4+4HZoJIXsPl5CVe4gujWhk+KlY4jBNWmxUBIt/cx5vTB7BsC6t+HJrPje98JXbYTVad999N3fffbfbYQRNVu4hhjy8gG37iujeOt7tcOqNJQ5j8PS2mnXrCEZ2S2JT7mEemLeOl5d953ZYjc7SpUtZunSp22EEzartB1CFey85jZ+c3dXtcOqNJQ5jfNxxYU8SYiL51xfbuPedtUx5dRWVlfbg3FTv6LovE0d0IqGRD/rzZYnDGB+DO7Vi5X0XkHH/BXROiuXdNbs4//FFlJZXuh2aaWAqK5XdB48wtEsrmkWEux1OvbLEYUw1EmKjeO+X53B2j2S27Sti/DNfsHZngdthmQbkZ6+uQhXGD2jvdij1zhKHMScRExXOzFs8ffLX7zrIpU9/zuSXVlJYYmua11ZqaiqpqdUtwRN61uYUEBcVzuX9m17isHEcxtQgPEyYdesIPsrcw/RPN/P+ut10/SSOX1/UC88KxyYQL7/8stshBMVXW/PZkV/ML87vTvPoxrMkrL/sjsMYP1zQpw1vTB5Bj9bxPLtoM7fPyqC8wp57NFUrvvNMVXPFwA4uR+IOSxzG+ElEeOHmIURHhvFORg5XPruErFybXTcQU6dOZerUqW6HUWf/98FGkuOb0TWl6Yzd8GWJw5gApLWKJeP+CxnUMYFvdhYw5onFrMk+4HZYISMjI4OMjAy3w6iT/YWlqMLAjgluh+IaSxzGBCg6Mpy3bjuTx67pD8DV05das1UTsnVfIQAT0tNcjsQ9jiYOEblYRDaKSJaI3FXN/t4islRESkTkTn/KikgrEflIRDZ5f9qsdMYVVw9O5fphHSktr2Tr3kK3wzH1ZJv3d90lpenMTVWVY4lDRMKBZ4CxQB/gOhHpU+WwfOAXwGMBlL0LWKiqPYCF3s/GuOL6oR0B2JR72OVITH3ZureQ8DAhLbFxL9ZUE78Sh4hMqcU3+6FAlqpuUdVS4HVgvO8BqpqrqsuBsgDKjgde9L5/EbgiwLiMCZq2LaMByDtU4nIkoaFnz5707NnT7TDqZMveQlITY4iKaLot/f6O42gLLBeRVcALwAd66pVvOgA7fD5nA8P8vF5NZduo6i4AVd0lIq2rO4GITAImAXTs2NHPyxoTmMTYKMLEM7W2ObUZM2a4HUKtZeUe4sF317Nq+/4mv26LXylTVe8FegD/BG4GNonIn0SkWw3Fqhsd5e9scXUp6zlYdYaqpqtqekpKSiBFjfFbeJiQFN+MaYs2M+ThBWzaY91zG6tPv93Lp9/m0bNNc64Z3HQfjEMAzzi8dxi7va9yIBF4Q0QePUmRbMD3v24qkOPn5Woqu0dE2gF4f+b6eU5jHPHA5acztm878g6VcMFfF/OvL7a6HVKDNWnSJCZNmuR2GLVSUFSKCPzn1hFc0q9prPR3Mv4+4/iFiKwEHgW+APqq6k+BwcBVJym2HOghIl1EJAq4FpjrZ1w1lZ0L3OR9fxMwx89zGuOIcX3b8fR1A3n+xnQAHpiXyZdb9rkcVcP07bff8u2337odRq3sLyqjZUwkYWE21Yy/zziSge+p6nEr26hqpYhcWl0BVS0XkSnAB0A48IKqrhORyd7900WkLbACaAFUishUoI+qHqyurPfUjwCzReRHwHbgmgDqa4xjxvRpw3MTB3PrSyv54czl3HZedwBio8L5wbBOTfphamOwv6iUxCa05kZN/E0cXaomDRF5SVUnqur6kxVS1fnA/Crbpvu8342nGcqvst7t+4DRfsZtTL266PS2PHP9IH7+2ir+74ONx7Zn5R7m4Sv7uhiZqavcgyUkx1viAP8Tx+m+H7zjLAYHPxxjQt8l/dpx4eljqVRFFc76y8f8Z0U2ew56el5NHNGJc3tah41QsiXvMF9ty+eawY1jSvi6qvHeWUTuFpFDQD8ROeh9HcLzQNqeLRhzEpHhYTSLCCc6MpxXfzKc09q3IOdAMV9u3cdNL3zFvibafXfAgAEMGDDA7TAC9uSCTQAM6dLK5UgaBjn1cAwQkT+r6t31EI8j0tPTdcWKFW6HYQxvrcrmV7NXM7JbEq/+ZLjb4Rg/7C8sZeCDHzXJ35mIrFTV9KrbT3XH0dv79j8iMqjqy5FIjWnEvjcolcGdElmyeR8frNvtdjjGD9M/3Qw07dlwqzrVM45f4Rl9/Xg1+xQ4P+gRGdPITb9hMEMeXsCbK7O56PS2bodTr2644QYgNFYCrKhUMnbsZ9X2/STHR3HHBb3cDqnBqDFxqOokEQkD7lXVL+opJmMatZTmzbisf3s+Xr8HVW1SS9BmZ2e7HYLf3lu7iymvfg3A+AHtbfyGj1N2LFfVSqrMXmuMqZuereMpLK3gL+9v5EhZhdvhmGqs/G4/YQKv/mQYD15xhtvhNCj+jkj6UESukqb01cgYB91yVhfCxNN+frTHjmlY/vXFNpLimzGyWzItoiPdDqdB8Tdx/Ar4D1BytEuuiBx0MC5jGrX4ZhF8+uvziAwXMnfZ/0oNzdG7wLO7J7scScPk1wBAVW3udCDGNDVprWK5pG87lm/b73Yo9WbEiBFuh+CXA0WeJYLSO9u4jer4lThE5Jzqtqvq4uCGY0zT0i0lnncycigqLSc2yt+JHELXn//8Z7dD8Mv+olIAEmKtiao6/v5L/bXP+2g8K/StxLrjGlMnPdt6bubnrc5hwhBbcKyhmLvas4qDJY7q+dtUdZnvZxFJwzPFujGmDkZ2SwLgbws20SwinHF92zXqWXSvusqzCsObb77pciQ125pXCMDAtKa90t/J1PbeOBuw/mnG1FHz6Ej+cFkf/jAvk6mzMoiODOfiMxrvoMB9+xruOiU7DxTzu7e/oaSskrU7CxjduzUxUeFuh9Ug+buQ09Mi8pT39XfgM2C1s6EZ0zTcfGYXPvvNeQC8u8bfRTJNsC3+No9FG/M4Ul7Bae1acE26zYR7Mv7ecfjOEFgOvGYjyY0JnrRWsSTGRvLe2t0cKasgOtK+6da3V778jshw4Y3JIwm3UeI18uuOQ1VfBF4DvgbW4Fna1RgTRA9f2ZeKSuU/K0NnWo7GoqJSycw5SHJ8M0safvC3O+444DlgMyBAFxG5VVXfczI4Y5qS83u3BuBfX2xl4vBOLkfjjNGjG+binTkHiqlU+OXoHm6HEhL8bap6AjhPVbMARKQb8F/AEocxQRIdGc51Q9N4Y2U2K7blM7BjYqP79nvfffe5HUK1dh4oBiA1MdblSEKDv/3+co8mDa8teFYBNMYE0fCuSZRVKFdPX8qf56+npNwmQKwPR0eKJ8bZuA1/+Js41onIfBG5WURuAuYBy0XkeyLyvZMVEpGLRWSjiGSJyF3V7BdvT60sEVlzdHEoEeklIhk+r4MiMtW77w8istNn37jAq21Mw3Rpv/a8ddtI2reM5vnPtzL19YxGNXvu2LFjGTt2rNthnKCg+OhI8SiXIwkN/iaOaGAPcC4wCsgDWgGXAZdWV0BEwoFngLFAH+A6EelT5bCxQA/vaxIwDUBVN6rqAFUdAAwGioC3fcr99eh+VZ3vZx2MafDCw4RBHRN5bdJwUhNjeG/tbn7y78az7HFxcTHFxcVuh3FMRaWSc6CYHfmemBJi7I7DH/6OHL+lFuceCmSp6hYAEXkdGA9k+hwzHvi3ehY+XyYiCSLSTlV3+RwzGtisqt/VIgZjQlKnpDjemDyS8c98zrocmz3XKfe+s5bXvtoOQExkOLE24M8v/vaq6gL8HOjsW0ZVL6+hWAdgh8/nbGCYH8d0AHwTx7V4ugL7miIiN+IZX3KHqjad6UVNk9G2ZTS3nNmFR97bwP7CUhLjrBkl2NbuLOC0di24eWQnuiTHN6nVGOvC36aqd4BtwNN41h8/+qpJdb8BDeQYEYkCLsezFshR04BuwAA8CabaOERkkoisEJEVeXl5pwjVmIZpUEfPXEnLtjTcqTpC1ea8w3yzs4D0TolMGNKRoV1sCnV/+dsd94iqPhXgubOBNJ/PqUDV+RROdcxYYJWq7jm6wfe9iPwDeLe6i6vqDGAGQHp6etWEZUxIGNgxgfhmESzetJexfdu5HU6dXXpptY9EXbHSuw7K6NNauxxJ6PE3cfxNRH4PfAiUHN2oqqtqKLMc6OFt5tqJp8np+irHzMXT7PQ6nmasgirPN66jSjNVlWcgVwJr/ayDMSEnMjyMkd2SeHdNDnsOHgGgXctoHhx/BmEhOMbjzjvvdDsEAJZvy+c3b65BBM6yVf4C5m/i6AtMxLP+RqV3m1LDehyqWi4iU4APgHDgBVVdJyKTvfunA/OBcUAWnp5Txx7Ci0gscAFwa5VTPyoiA7zX31bNfmMalYkjOrHn4BHyDpVQWFLOxxtyKSwp56weKVw92Cbiq413vett3HpONyLCG+809k4RT4emUxwksgHop6qlzocUfOnp6bpiRePp0miarv2FpVw9fQm7Co5QVFrBZ785j7RWoTPaedSoUQAsWrTItRhUlZ73vkevts159+dnuxZHKBCRlaqaXnW7v6l2NZAQ1IiMMQFLjIti4R2jmHnLUAD++G4mCzL3UF5ReYqS5qjNeYWUVSgdEmLcDiVk+dtU1QbYICLLOf4ZR03dcY0xDhnapRWje7fmo8w9fJS5h79dO4DxAzq4HVZI+G6fZ3W/W8/t5nIkocvfxPF7R6MwxgTsmR8MYnt+EROeW8qijXlc0redtdf74bt9RQB0TopzOZLQ5e/I8U+dDsQYE5joyHB6tmnOyO7JvP31Tj7K3MPHd5xL6xbRbofWoK3OPkDzZhEkxtr0IrVVY+IQkUOcOGgPPAP3VFVbOBKVMcZvv7moF51axfLsos3cP2cd0ycOdjukk/r+97/v6vVLyiuYk5FD99Y2Srwuakwcqtq8vgIxxtROp6Q4fn1RL+Zk5LB4Ux4/fnE5gzolctuo7m6HdoLbbrvN1esfnczwsn7tXY0j1FmDqDGNgIjwt2sH0KN1PGt3HuSJD7+lqLTc7bBOUFRURFFRkSvX3pFfxJgnPK3uZ/WwQX91YYnDmEYivXMr5kw5i0ev7kd5pfLPz7ayescBt8M6zrhx4xg3zp0ldL72/re4vH97+qe2dCWGxsIShzGNTHrnROKbRfD4R98y/pkv2F8YkuN2g+pIWQW//s9qAP5yVT/rfVZH9l/PmEYmNiqCj+88l79dOwCAP8xb525ADcC6nAJKyisZ2DGBGFtzo84scRjTCLVuHs3l/dvTKSmWORk57Mh357lCQ/HUwiwAHrumv8uRNA6WOIxppESE+y/1rNb8x3czT3F047W74AiffutZkyctMXTm9WrI/B05bowJQaNPa0ObFs34bFMeZRWVRLrctn/zzTfX+zW3eacYeelHQ4mKsO/KwWD/FY1p5O4ZdxpHyirJyj3sdijcfPPN9Z48tnunGOnUyqYYCRZLHMY0ch29067v3F/sciSwd+9e9u7dW6/X3LavkPAwoX2CTcUSLNZUZUwjl+pt19+x3/0H5FdffTVQf+txqCrPLtpMamKMdcENIvsvaUwjlxwfRXJ8Mx6Yl8kH63a7HU69yjvsWQViaJdWLkfSuFjiMKaRExF+eFZnAH7x2teUNYFFn0rKK7j1pRXc9MJyAC7rb3NTBZMlDmOagNtGdWfKed0pKa/kw3V73A7HcZv2HOaDdXsIE7j49LYM7pTodkiNiiUOY5qIX47pAcDybfkuR+K87d4Bj49e3Y/pEwfTItrW3ggmRx+Oi8jFwN+AcOB5VX2kyn7x7h8HFAE3q+oq775twCGgAig/umC6iLQCZgGdgW3A91V1v5P1MKYxiAwPY2S3JN5Ymc2q7fv54/gzGJCWUK8x/PSnP62X6xxNHEd7lJngcuyOQ0TCgWeAsUAf4DoR6VPlsLFAD+9rEjCtyv7zVHXA0aThdRewUFV7AAu9n40xfrj13G4M75rEht2H+OnLKzlcUr9Tr0+YMIEJEyY4eo3DJeXMzcihVVwUze1OwxFONlUNBbJUdYuqlgKvA+OrHDMe+Ld6LAMSRKTdKc47HnjR+/5F4IogxmxMo3ZuzxSevymdYV1asavgCB9l1m8vqx07drBjxw5Hr/Hwf9eTuesg3VvHO3qdpszJxNEB8P0Xku3d5u8xCnwoIitFZJLPMW1UdReA92froEZtTBMwY6LnJj7nwJF6ve7EiROZOHGio9fYXVCMCDxz/SBHr9OUOZk4qlvQt+r65TUdc6aqDsLTnPUzETknoIuLTBKRFSKyIi8vL5CixjR6MVHhtIyJ5P8+2NjoxnYUFJcxslsSKc2buR1Ko+Vk4sgG0nw+pwI5/h6jqkd/5gJv42n6AthztDnL+zO3uour6gxVTVfV9JSUlDpWxZjG544LewKwILNxdc8tKC6jZYw923CSk4ljOdBDRLqISBRwLTC3yjFzgRvFYzhQoKq7RCRORJoDiEgccCGw1qfMTd73NwFzHKyDMY3WjSM6MyAtgW8bwOSHwVJQXMbmvEJLHA5zrDuuqpaLyBTgAzzdcV9Q1XUiMtm7fzowH09X3Cw83XFv8RZvA7zt6a1LBPCqqr7v3fcIMFtEfgRsB65xqg7GNHZJcVEs3JDLroJi2rWMcTucOlu2ZR8A3VLswbiTHB3Hoarz8SQH323Tfd4r8LNqym0Bql2qS1X3AaODG6kxTdMl/dqxcEMuW/MK6yVx3HHHHY6e/+hKh1cPTnX0Ok2dzY5rTBM2sKNnKo7pi7cwsGOi4+txX3bZZY6ef3t+Ec2jI6ypymE25YgxTVi7ltHERIaz+Ns83ljp7PgKgI0bN7Jx48agna+wpJzcQ0eOvTbnHaZjq1i8zdzGIXbHYUwTFh0Zzle/G83ghxZw35x1/GBYJ8LCnPuje+uttwLBWY8jv7CUkY8s5EjZ8bP9XtL3VGOITV1Z4jCmiWseHck1g1N55cvtPLlwE7+6oKfbIfklK/cwR8oq+dFZXeiS/L9lYc/tad3vnWaJwxjDlPO788qX23nh860hkTi27i1kwoylANwwvNNxicM4z55xGGNo1zKG31zci8Ml5RSV1u/Eh7Xx2aY8VOHCPm1sBlwXWOIwxgDQtkU0ALsL6nf+qkCVllfy7CebiYoI47mJgwl38JmMqZ41VRljAJ/EcfAIXR0aQHfvvffW+RxfbN7L7oNH6NE63npPucQShzEGgDYtPYljz0Hn7jjGjBlTp/Kqyheb9gLw8o+HBSMkUwvWVGWMAf53x3H7rNUcKatw5BoZGRlkZGTUuvzLy77j+c+3Et8sgtY2+61rLHEYYwCIaxbBwI4JgKfXkhOmTp3K1KlTa10+c9chAF758TBrpnKRJQ5jzDH3XnIa4GxzVV1k7y+if2pL+tfzWunmeJY4jDHHtG7uaa6auWSbu4GcxPb8ItKs+63rLHEYY45JTYyhZUwkizbmsfK7/W6Hc8zWvYU8uyiLnAPFljgaAEscxphjRIQ7vSsD3vD8l5RVVJ6iRP14auEmHn1/IxWVyiDvjL7GPdYd1xhznBuGd6KwtIJH3tvAba+s4h83pgft3H/6059qVe67fYUM69KKl388jMhw+77rNvsNGGOOIyJcP6wjcVHhfLIhl3mrc9hVUByUc48cOZKRI0cGXG7H/mI6JcVa0mgg7LdgjDlBi+hInrpuIOWVys9f+5rfvb02KOddsmQJS5YsCajMkbIK8g6VkJZozzYaCkscxphqnd+7NZ/+ehSX9G3Hxxty+f2cuiePe+65h3vuuSegMtn7PcvB2kPxhsMShzGmWiJCp6Q4fnR2FwBeXPodq3ccqPc4duR7msnSWjm/Jrrxj6OJQ0QuFpGNIpIlIndVs19E5Cnv/jUiMsi7PU1EPhGR9SKyTkR+6VPmDyKyU0QyvK9xTtbBmKZuUMdE/nXLEADGP/PFsTuA+rJwwx4Aa6pqQBxLHCISDjwDjAX6ANeJSJ8qh40Fenhfk4Bp3u3lwB2qehowHPhZlbJ/VdUB3td8p+pgjPEY1TOFBy4/HYAJzy0j50BwHpb7Y9HGPACS421uqobCyTuOoUCWqm5R1VLgdWB8lWPGA/9Wj2VAgoi0U9VdqroKQFUPAeuBDg7GaoypgYhw3dCODOmcyM4DxbyTsdPxa1ZWKkfKKthz8Ag3j+zs6FroJjBOJo4OwA6fz9mc+Mf/lMeISGdgIPClz+Yp3qatF0TERgMZUw+iIsL4z+SRJMVF8ej7G1HVgM/x5JNP8uSTT/p17Ni/fUbv+96nrELp0caZ9UFM7TiZOKr7elD1X1qNx4hIPPAmMFVVD3o3TwO6AQOAXcDj1V5cZJKIrBCRFXl5eQGGbow5mXF92wHw8H/XU1kZWPIYMGAAAwYMOOVxBcVlbNxziNG9W/O7cadxWf/2tQnVOMTJxJENpPl8TgVy/D1GRCLxJI1XVPWtoweo6h5VrVDVSuAfeJrETqCqM1Q1XVXTU1JS6lwZY4zHzWd2BuD5z7eyantg81ktWLCABQsWnPK4D9btBuDqwan85JyutIiODDhO4xwnE8dyoIeIdBGRKOBaYG6VY+YCN3p7Vw0HClR1l3gm2v8nsF5Vn/AtICLtfD5eCQRnZJIxxi/dUuJZctf5APz43yt45pMsv8s+9NBDPPTQQ6c87sst+QAMtHmpGiTHEoeqlgNTgA/wPNyerarrRGSyiEz2HjYf2AJk4bl7uM27/UxgInB+Nd1uHxWRb0RkDXAecLtTdTDGVK9dy2juvLAnLaIjmbZoc8BNVjXZe7iEjzJ3k94pkbbe5WxNw+LoJIferrLzq2yb7vNegZ9VU+5zqn/+gapODHKYxpgAiQhTzu9BQXEZ//hsK59n7eWcnsFpEr7t5VUcPFJO99b2QLyhspHjxphamzi8MwDZ+4M3rmNT7iF6t23O77yrEZqGxxKHMabW2iV4mpLyDpUE5Xwrv9vP/qIyLuvfnub2QLzBsvU4jDG1FhkeRqu4KJ76eBPPLvI8JL9tVHd+OaZHtcc/99xzNZ4vM6cAgOFdk4IbqAkqSxzGmDp56IozWJ19AIAP1+3hn59v4efnd692pHevXr1qPFdJuWfFwZ424K9Bs8RhjKmTcX3bHRsUeLC4nNe+2s7K7fsZ0rnVCcfOmzcPgMsuu6zacx1NHM0iwh2K1gSDPeMwxgTNdUM943lP9szj8ccf5/HHq53sAYCSsgpEIDLc5qVqyCxxGGOCJsk7g+2hI2W1Kl9SXkmziDA8Y4BNQ2WJwxgTNM2jPa3fh46U16q8J3FYM1VDZ4nDGBM08VERiMDB4trecVTQLML+LDV09hsyxgRNWJjQvFkE+UWltSpfUlZJs0j7s9TQWa8qY0xQdU2J561VO3lw/BknPKt46aWXTlquvKKSDbsPWVNVCLDUbowJqq4pcRSVVvD1jgMn7EtLSyMtLe3EQsCf39tA5q6Dx56TmIbLEocxJqhuH9MTgKmvZ1BeUXncvlmzZjFr1qxqy23cfQiAx6/p72yAps4scRhjgio1MYa0VjFszy9iU+7h4/ZNmzaNadOmnVBmxbZ8Ps/ay6X92tE1xUaNN3SWOIwxQSUiPH/jEACeWrjJrzJvf70TgEv72RKxocAShzEm6Hq2iSc2Kpyvtx845bH7Dpfw/trd9O3QkovPaOt8cKbOLHEYY4JORJg4vBO7Dx5hSdbeGo+9ffZq9hWW0i0lrp6iM3VlicMY44gbR3YG4D8rs5m7Ooft+4qqPW5z7mG6JsfxxyvOqMfoTF1YvzdjjCM6JMTQLSWOt7/eydtf72Ro51a88cYbxx1TXlHJ7oNHuG1UN1rYwk0hwxKHMcYx7/zsTPYcLOGZT7JYkLmHpKSk4wYF7j54hIpKpUNCjItRmkA52lQlIheLyEYRyRKRu6rZLyLylHf/GhEZdKqyItJKRD4SkU3en4lO1sEYU3vNoyPp3jqegR0TOFRSzlPTnuehJ57l/bW7eX/tbuZk5ACQmhjrcqQmEI7dcYhIOPAMcAGQDSwXkbmqmulz2Figh/c1DJgGDDtF2buAhar6iDeh3AX81ql6GGPqrkfr5gA8+NdnOVxSTtvcTsf2iUC31vZgPJQ42VQ1FMhS1S0AIvI6MB7wTRzjgX+rqgLLRCRBRNoBnWsoOx4Y5S3/IrAISxzGNGj9UltyZvck3izxTLf+64t6cV6v1gC0iImgXUtrqgolTjZVdQB2+HzO9m7z55iayrZR1V0A3p+tgxizMcYBcc0ieOXHw49NYJjeKZE+7VvQp30La6YKQU4mjuqW8FI/j/GnbM0XF5kkIitEZEVeXl4gRY0xDumaEkfrFtH0T0twOxRTB04mjmzAdxrMVCDHz2NqKrvH25yF92dudRdX1Rmqmq6q6SkpKbWuhDEmeFrGRNI1OY7oSJs6PZQ5mTiWAz1EpIuIRAHXAnOrHDMXuNHbu2o4UOBtfqqp7FzgJu/7m4A5DtbBGBNE8+fPZ/78+W6HYerIsYfjqlouIlOAD4Bw4AVVXScik737pwPzgXFAFlAE3FJTWe+pHwFmi8iPgO3ANU7VwRgTXLGx9jyjMRBPh6bGLT09XVesWOF2GMY0ec8++ywAt912m8uRGH+IyEpVTa+63eaqMsbUm9mzZzN79my3wzB1ZInDGGNMQCxxGGOMCYglDmOMMQGxxGGMMSYgTaJXlYjkAd/VsngyUPMSZo2P1blpsDo3DXWpcydVPWEEdZNIHHUhIiuq647WmFmdmwarc9PgRJ2tqcoYY0xALHEYY4wJiCWOU5vhdgAusDo3DVbnpiHodbZnHMYYYwJidxzGGGMCYonDS0QuFpGNIpLlXcu86n4Rkae8+9eIyCA34gwmP+r8A29d14jIEhHp70acwXSqOvscN0REKkTk6vqMzwn+1FlERolIhoisE5FP6zvGYPPj33ZLEZknIqu9db7FjTiDRUReEJFcEVl7kv3B/fulqk3+hWfq9s1AVyAKWA30qXLMOOA9PKsTDge+dDvueqjzSCDR+35sU6izz3Ef45n2/2q3466H33MCkAl09H5u7Xbc9VDne4C/eN+nAPlAlNux16HO5wCDgLUn2R/Uv192x+ExFMhS1S2qWgq8Doyvcsx44N/qsQxIOLoSYYg6ZZ1VdYmq7vd+XIZnJcZQ5s/vGeDnwJucZHXJEONPna8H3lLV7QCqGur19qfOCjQXEQHi8SSO8voNM3hUdTGeOpxMUP9+WeLw6ADs8Pmc7d0W6DGhJND6/AjPN5ZQdso6i0gH4Epgej3G5SR/fs89gUQRWSQiK0XkxnqLzhn+1PnvwGl4lqT+BvilqlbWT3iuCOrfL8dWAAwxUs22qt3N/DkmlPhdHxE5D0/iOMvRiJznT52fBH6rqhWeL6Mhz586RwCDgdFADLBURJap6rdOB+cQf+p8EZABnA90Az4Skc9U9aDDsbklqH+/LHF4ZANpPp9T8XwTCfSYUOJXfUSkH/A8MFZV99VTbE7xp87pwOvepJEMjBORclV9p14iDD5//23vVdVCoFBEFgP9gVBNHP7U+RbgEfU8AMgSka1Ab+Cr+gmx3gX175c1VXksB3qISBcRiQKuBeZWOWYucKO3d8JwoEBVd9V3oEF0yjqLSEfgLWBiCH/79HXKOqtqF1XtrKqdgTeA20I4aYB//7bnAGeLSISIxALDgPX1HGcw+VPn7XjusBCRNkAvYEu9Rlm/gvr3y+44AFUtF5EpwAd4emS8oKrrRGSyd/90PD1sxgFZQBGebywhy8863w8kAc96v4GXawhPEOdnnRsVf+qsqutF5H1gDVAJPK+q1XbrDAV+/p4fBGaKyDd4mnF+q6ohO2uuiLwGjAKSRSQb+D0QCc78/bKR48YYYwJiTVXGGGMCYonDGGNMQCxxGGOMCYglDmOMMQGxxGGMMSYgljiMqYGIJHlnjc0Qkd0istP7/oCIZDpwvT+IyJ0Bljl8ku0zG8PsvqbhscRhTA1UdZ+qDlDVAXjmr/qr9/0APGMeaiQiNlbKNDqWOIypvXAR+Yd3PYcPRSQGwDtZ4J+861r8UkQGi8in3gkEPzg6K6mI/EJEMr3rI7zuc94+3nNsEZFfHN0oIr8SkbXe19SqwXhHBf/de87/Aq2drb5pquzbkDG11wO4TlV/IiKzgauAl737ElT1XBGJBD4FxqtqnohMAB4GfgjcBXRR1RIRSfA5b2/gPKA5sFFEpgH98Iz2HYZnpPOXIvKpqn7tU+5KPFNn9AXa4Flj4wUnKm6aNkscxtTeVlXN8L5fCXT22TfL+7MXcAae2VfBMwXG0TmC1gCviMg7wDs+Zf+rqiVAiYjk4kkCZwFveyciRETeAs4GfBPHOcBrqloB5IjIx3WvojEnssRhTO2V+LyvwDMl+VGF3p8CrFPVEdWUvwTPH/vLgftE5PSTnDeC6qfFro7NIWQcZ884jHHWRiBFREYAiEikiJwuImFAmqp+AvwGz/Kt8TWcZzFwhYjEikgcnmapz6o55loRCfc+RzkvyHUxBrA7DmMcpaql3i6xT4lISzz/zz2JZ62Ll73bBE9vrQMnWzxKVVeJyEz+t17E81WebwC8jWdhom+85/80yNUxBrDZcY0xxgTImqqMMcYExBKHMcaYgFjiMMYYExBLHMYYYwJiicMYY0xALHEYY4wJiCUOY4wxAbHEYYwxJiD/D/4E5mJpS9RgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "feature = np.random.normal(size=1000)\n",
    "y = (feature >= 0.7).astype(int)\n",
    "thresholds = np.arange(0.0, 1, 0.001) \n",
    "gini_impurities = []\n",
    "for thresh in thresholds:\n",
    "    y_a = y[feature <= thresh]\n",
    "    y_b = y[feature >= thresh]\n",
    "    impurity = compute_impurity(y_a, y_b)\n",
    "    gini_impurities.append(impurity)\n",
    "\n",
    "best_thresh = thresholds[np.argmin(gini_impurities)]\n",
    "print(f\"Impurity is minimized at a threshold of {best_thresh:.02f}\")\n",
    "plt.plot(thresholds, gini_impurities)\n",
    "plt.axvline(best_thresh, c='k', linestyle='--')\n",
    "plt.xlabel('Threshold')\n",
    "plt.ylabel('Impurity')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn's Decision tree classifier can derive conditional logic from numeric data. Lets now train `clf` on the numeric wine data that we've introduced in the previous section.\n",
    "\n",
    "**Listing 22. 48. Training a Decision tree on numeric data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|--- proline <= 755.00\n",
      "|   |--- od280/od315_of_diluted_wines <= 2.11\n",
      "|   |   |--- hue <= 0.94\n",
      "|   |   |   |--- flavanoids <= 1.58\n",
      "|   |   |   |   |--- class: 2\n",
      "|   |   |   |--- flavanoids >  1.58\n",
      "|   |   |   |   |--- class: 1\n",
      "|   |   |--- hue >  0.94\n",
      "|   |   |   |--- color_intensity <= 5.82\n",
      "|   |   |   |   |--- class: 1\n",
      "|   |   |   |--- color_intensity >  5.82\n",
      "|   |   |   |   |--- class: 2\n",
      "|   |--- od280/od315_of_diluted_wines >  2.11\n",
      "|   |   |--- flavanoids <= 0.80\n",
      "|   |   |   |--- class: 2\n",
      "|   |   |--- flavanoids >  0.80\n",
      "|   |   |   |--- alcohol <= 13.17\n",
      "|   |   |   |   |--- class: 1\n",
      "|   |   |   |--- alcohol >  13.17\n",
      "|   |   |   |   |--- color_intensity <= 4.06\n",
      "|   |   |   |   |   |--- class: 1\n",
      "|   |   |   |   |--- color_intensity >  4.06\n",
      "|   |   |   |   |   |--- class: 0\n",
      "|--- proline >  755.00\n",
      "|   |--- flavanoids <= 2.17\n",
      "|   |   |--- malic_acid <= 2.08\n",
      "|   |   |   |--- class: 1\n",
      "|   |   |--- malic_acid >  2.08\n",
      "|   |   |   |--- class: 2\n",
      "|   |--- flavanoids >  2.17\n",
      "|   |   |--- magnesium <= 135.50\n",
      "|   |   |   |--- class: 0\n",
      "|   |   |--- magnesium >  135.50\n",
      "|   |   |   |--- class: 1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "from sklearn.datasets import load_wine\n",
    "X, y = load_wine(return_X_y=True)\n",
    "clf.fit(X, y)\n",
    "feature_names = load_wine().feature_names\n",
    "text_tree = export_text(clf, feature_names=feature_names)\n",
    "print(text_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The printed tree is large because it is **deep**. In machine learning, tree **depth** equals the number of nested if/else statements required to capture the logic within the tree. In Scikit-Learn, we can limit the depth of a trained tree using the `max_depth` hyperparameter.\n",
    "\n",
    "**Listing 22. 49. Training a tree with limited depth**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "|--- proline <= 755.00\n",
      "|   |--- od280/od315_of_diluted_wines <= 2.11\n",
      "|   |   |--- class: 2\n",
      "|   |--- od280/od315_of_diluted_wines >  2.11\n",
      "|   |   |--- class: 1\n",
      "|--- proline >  755.00\n",
      "|   |--- flavanoids <= 2.17\n",
      "|   |   |--- class: 2\n",
      "|   |--- flavanoids >  2.17\n",
      "|   |   |--- class: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf = DecisionTreeClassifier(max_depth=2)\n",
    "clf.fit(X, y)\n",
    "text_tree = export_text(clf, feature_names=feature_names)\n",
    "print(text_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The depth at which a feature appears within the tree is an indicator of its relative importance. That depth is determined by the Gini Impurity. Hence, the Gini Impurity can be leveraged to compute an importance score. The importance scores across all features are stored within the `feature_importances_` attribute.\n",
    "\n",
    "**Listing 22. 50. Printing the feature importances**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         0.         0.         0.         0.         0.\n",
      " 0.117799   0.         0.         0.         0.         0.39637021\n",
      " 0.48583079]\n"
     ]
    }
   ],
   "source": [
    "print(clf.feature_importances_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the features receive a score of `0` because they are not represented in the trained tree. Lets rank the remaining features based on their importance score.\n",
    "\n",
    "**Listing 22. 51. Ranking relevant features by importance**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'proline' has an importance score of 0.49\n",
      "'od280/od315_of_diluted_wines' has an importance score of 0.40\n",
      "'flavanoids' has an importance score of 0.12\n"
     ]
    }
   ],
   "source": [
    "for i in np.argsort(clf.feature_importances_)[::-1]:\n",
    "    feature = feature_names[i]\n",
    "    importance = clf.feature_importances_[i]\n",
    "    if importance == 0:\n",
    "        break\n",
    "    \n",
    "    print(f\"'{feature}' has an importance score of {importance:0.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tree-based feature ranking can help draw meaningful insights from our data. We'll emphasize this point by exploring the serious problem of cancer diagnosis.\n",
    "\n",
    "### 22.2.1 Studying Cancerous Cells Using Feature Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn includes a cancer-cell dataset. Below, we'll import that data from `sklearn.datasets`.\n",
    "\n",
    "**Listing 22. 52. Importing Scikit-Learn’s cancer-cell dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cancer dataset contains the following 2 classes:\n",
      "['malignant' 'benign']\n",
      "\n",
      "It contains these 30 features:\n",
      "['mean radius' 'mean texture' 'mean perimeter' 'mean area'\n",
      " 'mean smoothness' 'mean compactness' 'mean concavity'\n",
      " 'mean concave points' 'mean symmetry' 'mean fractal dimension'\n",
      " 'radius error' 'texture error' 'perimeter error' 'area error'\n",
      " 'smoothness error' 'compactness error' 'concavity error'\n",
      " 'concave points error' 'symmetry error' 'fractal dimension error'\n",
      " 'worst radius' 'worst texture' 'worst perimeter' 'worst area'\n",
      " 'worst smoothness' 'worst compactness' 'worst concavity'\n",
      " 'worst concave points' 'worst symmetry' 'worst fractal dimension']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "data = load_breast_cancer()\n",
    "feature_names = data.feature_names\n",
    "num_features = len(feature_names)\n",
    "num_classes = len(data.target_names)\n",
    "print(f\"The cancer dataset contains the following {num_classes} classes:\")\n",
    "print(data.target_names)\n",
    "print(f\"\\nIt contains these {num_features} features:\")\n",
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains 30 different features. Lets rank these features by importance.\n",
    "\n",
    "**Listing 22. 53. Ranking tumor features by importance**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'worst radius' has an importance score of 0.70\n",
      "'worst concave points' has an importance score of 0.11\n",
      "'worst texture' has an importance score of 0.06\n",
      "'mean concave points' has an importance score of 0.04\n",
      "'mean texture' has an importance score of 0.03\n",
      "'worst smoothness' has an importance score of 0.01\n",
      "'worst area' has an importance score of 0.01\n",
      "'worst symmetry' has an importance score of 0.01\n",
      "'radius error' has an importance score of 0.01\n",
      "'mean symmetry' has an importance score of 0.01\n",
      "'symmetry error' has an importance score of 0.01\n"
     ]
    }
   ],
   "source": [
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(X, y)\n",
    "for i in np.argsort(clf.feature_importances_)[::-1]:\n",
    "    feature = feature_names[i]\n",
    "    importance = clf.feature_importances_[i]\n",
    "    if round(importance, 2) == 0:\n",
    "        break\n",
    "    print(f\"'{feature}' has an importance score of {importance:0.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Among the top-ranking features, _worst radius_ particularly stands out. Its high score suggests that the radius of the largest cell is an extremely important indicator of cancer. We can check this hypothesis by plotting histograms of the _worst radius_ measurements across the two classes.\n",
    "\n",
    "**Listing 22. 54. Plotting two _worst radius_ histograms**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEGCAYAAAB8Ys7jAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXbUlEQVR4nO3df7TVdZ3v8ee7MyaMOKECDkVzD94sTLGjAiqlC39kFi51WpKZTpgl3kkm4t5WobMkZN21hm7ZYKuaWdQ4knoBR2fStFIXpabjhJCkKBQl5zYkFxCLZJJR7H3/2N/DPcD5sc85+5zDB56PtVh77+/+/nh/+MrLz/mc7/fzjcxEklSeNwx2AZKk3jHAJalQBrgkFcoAl6RCGeCSVKg/GsiDjRgxIpubmwfykJJUvFWrVr2YmSP3Xj6gAd7c3MzKlSsH8pCSVLyI+D8dLXcIRZIKZYBLUqEMcEkq1ICOgUs6cLz22mts3LiRnTt3DnYpB4whQ4YwZswYDjnkkLrWN8Al9crGjRs5/PDDaW5uJiIGu5ziZSbbtm1j48aNjB07tq5tHEKR1Cs7d+7kqKOOMrwbJCI46qijevQTjQEuqdcM78bq6d+nAS5JhXIMXFJDNM+5v6H7a10wtdt1IoIrrriC2267DYBdu3YxevRoTj31VO67775Ot3v44Yf50pe+xH333ce9997Lc889x5w5cxpWe1dWr17NCy+8wAc+8IE+78sA309s2DCv23XGju1+Helgcthhh7FmzRpeeeUVhg4dykMPPcRb3vKWHu3jwgsv5MILL+ynCve1evVqVq5c2ZAAdwhFUtHe//73c//9td7/kiVLuOyyy3Z/t2LFCiZPnsxJJ53E5MmT+dnPfrbP9rfeeiszZ84E4Je//CWnnXYaEydOZO7cuQwbNgyo9dinTJnCJZdcwrhx47j88stpe5rZ/PnzmThxIieccAIzZszYvXzKlCl87nOfY9KkSbz97W/nRz/6Ea+++ipz585l2bJltLS0sGzZsj613QCXVLQPf/jDLF26lJ07d/L0009z6qmn7v5u3LhxPProozz11FPMnz+f66+/vst9zZo1i1mzZvHkk0/y5je/eY/vnnrqKRYuXMhzzz3H888/z+OPPw7AzJkzefLJJ3f/JNB+6GbXrl2sWLGChQsXcuONN/LGN76R+fPnc+mll7J69WouvfTSPrXdAJdUtBNPPJHW1laWLFmyz7DE9u3bmTZtGieccAKzZ8/m2Wef7XJfTzzxBNOmTQPgIx/5yB7fTZo0iTFjxvCGN7yBlpYWWltbAfjhD3/Iqaeeyvjx4/nBD36wxzE++MEPAnDKKafsXr+RDHBJxbvwwgv5zGc+s8fwCcANN9zAWWedxZo1a/jOd77Tp7tGDz300N3vm5qa2LVrFzt37uSTn/wkd911F8888wxXX331Hsdo26Zt/UYzwCUV76qrrmLu3LmMHz9+j+Xbt2/f/UvNW2+9tdv9nHbaadx9990ALF26tNv128J6xIgR7Nixg7vuuqvbbQ4//HBefvnlbterR11XoUTEcOCbwAlAAlcBPwOWAc1AK/ChzPxNQ6qSVJx6LvvrL2PGjGHWrFn7LP/sZz/L9OnT+fKXv8zZZ5/d7X4WLlzIFVdcwU033cTUqVN505ve1OX6w4cP5+qrr2b8+PE0NzczceLEbo9x1llnsWDBAlpaWrjuuuv6NA4ebb8x7XKliMXAjzLzmxHxRuCPgeuBlzJzQUTMAY7IzM91tZ8JEyakD3TomJcRqjRr167luOOOG+wyGur3v/89Q4cOJSJYunQpS5Ys4Z577hnQGjr6e42IVZk5Ye91u+2BR8SfAGcCVwJk5qvAqxFxETClWm0x8DDQZYBL0v5s1apVzJw5k8xk+PDh3HLLLYNdUpfqGUI5BtgK/GNEvAtYBcwCjs7MTQCZuSkiRvVfmZLU/8444wx++tOfDnYZdavnl5h/BJwM/F1mngT8B1D3PacRMSMiVkbEyq1bt/ayTEnS3uoJ8I3Axsz8cfX5LmqBvjkiRgNUr1s62jgzF2XmhMycMHLkPg9VliT1UrcBnpn/F/j3iHhHtegc4DngXmB6tWw6MLAj/ZJ0kKt3Mqu/Au6orkB5HvgYtfC/MyI+DvwKmNY/JUqSOlJXgGfmamCfS1io9cYlqa5LYXuinstmm5qaGD9+PJlJU1MTX/3qV5k8eXKvjjd37lzOPPNMzj333F5tPxicTlZSsYYOHcrq1asBeOCBB7juuut45JFHerWv+fPnN7CygeGt9JIOCL/73e844ogjdn/+4he/yMSJEznxxBP5/Oc/D0BrayvHHXccV199NccffzznnXcer7zyCgBXXnnl7lvhv/vd7zJu3Dje85738KlPfYoLLrgAgHnz5nHVVVcxZcoUjjnmGL7yla8McCv3ZIBLKtYrr7xCS0sL48aN4xOf+AQ33HADAA8++CDr169nxYoVrF69mlWrVvHoo48CsH79eq699lqeffZZhg8fvnvukzY7d+7kmmuu4Xvf+x6PPfYYe1/+vG7dOh544AFWrFjBjTfeyGuvvTYwje2AAS6pWG1DKOvWreP73/8+H/3oR8lMHnzwQR588EFOOukkTj75ZNatW8f69esBGDt2LC0tLUDH07yuW7eOY445hrFjxwLsM8Ph1KlTOfTQQxkxYgSjRo1i8+bN/d7OzjgGLumAcPrpp/Piiy+ydetWMpPrrruOa665Zo91Wltb95kWtm0IpU1380N1NK3sYLEHLumAsG7dOl5//XWOOuoo3ve+93HLLbewY8cOAH7961+zZUuH9xruY9y4cTz//PO7e+Z9fexZf7IHLqkhBmO2zLYxcKj1nBcvXkxTUxPnnXcea9eu5fTTTwdg2LBh3H777TQ1NXW7z6FDh/L1r3+d888/nxEjRjBp0qT+bEKf1DWdbKM4nWznnE5WpTkQp5Nts2PHDoYNG0Zmcu2113Lssccye/bsATl2T6aTdQhFkvbyjW98g5aWFo4//ni2b9++z1j6/sIhFEnay+zZswesx90X9sAl9dpADsEeDHr692mAS+qVIUOGsG3bNkO8QTKTbdu2MWTIkLq3cQhFUq+MGTOGjRs37nOnonpvyJAhjBkzpu71DXBJvXLIIYfsvltRg8MhFEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFqutW+ohoBV4GXgd2ZeaEiDgSWAY0A63AhzLzN/1TpiRpbz3pgZ+VmS3tngoxB1iemccCy6vPkqQB0pchlIuAxdX7xcDFfa5GklS3egM8gQcjYlVEzKiWHZ2ZmwCq11EdbRgRMyJiZUSsdNpJSWqceqeTfXdmvhARo4CHImJdvQfIzEXAIqg91LgXNUqSOlBXDzwzX6hetwD/AkwCNkfEaIDqdUt/FSlJ2le3AR4Rh0XE4W3vgfOANcC9wPRqtenAPf1VpCRpX/UMoRwN/EtEtK3/vzPz+xHxJHBnRHwc+BUwrf/KLN+GDfMGuwRJB5huAzwznwfe1cHybcA5/VGUJKl73okpSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFcoAl6RCGeCSVKhun0qv/ceGDfP6tP3YsX3bXtL+xR64JBXKAJekQtUd4BHRFBFPRcR91ecjI+KhiFhfvR7Rf2VKkvbWkx74LGBtu89zgOWZeSywvPosSRogdQV4RIwBpgLfbLf4ImBx9X4xcHFDK5MkdaneHvhC4LPAH9otOzozNwFUr6M62jAiZkTEyohYuXXr1r7UKklqp9sAj4gLgC2Zuao3B8jMRZk5ITMnjBw5sje7kCR1oJ7rwN8NXBgRHwCGAH8SEbcDmyNidGZuiojRwJb+LFSStKdue+CZeV1mjsnMZuDDwA8y8wrgXmB6tdp04J5+q1KStI++XAe+AHhvRKwH3lt9liQNkB7dSp+ZDwMPV++3Aec0viRJUj28E1OSCuVkVlIXmufc3/B9ti6Y2vB96uBkD1ySCmWAS1KhDHBJKpQBLkmFMsAlqVBehaJu9ceVGP3Bqzt0sLEHLkmFMsAlqVAGuCQVyjHwBtmwYd5glyDpIGMPXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmF6jbAI2JIRKyIiJ9GxLMRcWO1/MiIeCgi1levR/R/uZKkNvX0wP8TODsz3wW0AOdHxGnAHGB5Zh4LLK8+S5IGSLcBnjU7qo+HVH8SuAhYXC1fDFzcHwVKkjpW1xh4RDRFxGpgC/BQZv4YODozNwFUr6M62XZGRKyMiJVbt25tUNmSpLoCPDNfz8wWYAwwKSJOqPcAmbkoMydk5oSRI0f2skxJ0t56dBVKZv4WeBg4H9gcEaMBqtctjS5OktS5bh+pFhEjgdcy87cRMRQ4F/gCcC8wHVhQvd7Tn4UONh+ZJml/U88zMUcDiyOiiVqP/c7MvC8ingDujIiPA78CpvVjnZKkvXQb4Jn5NHBSB8u3Aef0R1GSpO55J6YkFaqeIRQV5OblP+/0u2//4v4BrERSf7MHLkmFMsAlqVAOoeiA0TzHISIdXOyBS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEJ5GaF2u/htd3T5/bd/cfkAVSKpHvbAJalQBrgkFcoAl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYUywCWpUAa4JBXKAJekQhngklSobiezioi3At8C/hT4A7AoM2+OiCOBZUAz0Ap8KDN/03+l9q8NG+YNdgk6SPTHsztbF0xt+D61/6unB74L+B+ZeRxwGnBtRLwTmAMsz8xjgeXVZ0nSAOk2wDNzU2b+pHr/MrAWeAtwEbC4Wm0xcHE/1ShJ6kCPxsAjohk4CfgxcHRmboJayAOjOtlmRkSsjIiVW7du7WO5kqQ2dQd4RAwD7gY+nZm/q3e7zFyUmRMyc8LIkSN7U6MkqQN1BXhEHEItvO/IzH+uFm+OiNHV96OBLf1ToiSpI/VchRLAPwBrM/PL7b66F5gOLKhe7+mXCtUw3T0yTVJZ6nkm5ruBvwCeiYjV1bLrqQX3nRHxceBXwLR+qVCS1KFuAzwzHwOik6/PaWw5kqR6eSemJBXKAJekQhngklQoA1ySCmWAS1KhDHBJKpQBLkmFMsAlqVAGuCQVygCXpEIZ4JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFaqeR6odEDZsmDfYJUhSQ9kDl6RCGeCSVCgDXJIKZYBLUqEMcEkqlAEuSYXqNsAj4paI2BIRa9otOzIiHoqI9dXrEf1bpiRpb/X0wG8Fzt9r2RxgeWYeCyyvPkuSBlC3AZ6ZjwIv7bX4ImBx9X4xcHFjy5Ikdae3Y+BHZ+YmgOp1VGcrRsSMiFgZESu3bt3ay8NJkvbW77/EzMxFmTkhMyeMHDmyvw8nSQeN3gb45ogYDVC9bmlcSZKkevR2Mqt7genAgur1noZVJKnHmufc3/B9ti6Y2vB9qrHquYxwCfAE8I6I2BgRH6cW3O+NiPXAe6vPkqQB1G0PPDMv6+SrcxpciySpB7wTU5IKZYBLUqEMcEkqlAEuSYU6YJ6J6TMv+9/Fb7ujy++//YvLB6gSSWAPXJKKZYBLUqEMcEkqlAEuSYUywCWpUAfMVSglunn5zwe7BEkFswcuSYUywCWpUAa4JBXKMXBJHeqPh0T0h4P5wRP2wCWpUAa4JBXKIRQ1jJNdaTAczM8DtQcuSYUywCWpUAa4JBXKAJekQhngklQoA1ySCtWnywgj4nzgZqAJ+GZmLmhIVR3wmZcHPi9D1P6ilEsTe90Dj4gm4GvA+4F3ApdFxDsbVZgkqWt9GUKZBPwiM5/PzFeBpcBFjSlLktSdvgyhvAX493afNwKn7r1SRMwAZlQfd0TEz/pwzP4yAnhxsIvoB/tZu5Z0+e3N9W+/n7WrYWxXWXrUrvhCn471Xzpa2JcAjw6W5T4LMhcBi/pwnH4XESszc8Jg19Fotqsstqss+0O7+jKEshF4a7vPY4AX+laOJKlefQnwJ4FjI2JsRLwR+DBwb2PKkiR1p9dDKJm5KyJmAg9Qu4zwlsx8tmGVDaz9eoinD2xXWWxXWQa9XZG5z7C1JKkA3okpSYUywCWpUAdVgEfELRGxJSLWtFt2ZEQ8FBHrq9cjBrPG3uikXfMi4tcRsbr684HBrLE3IuKtEfHDiFgbEc9GxKxqedHnrIt2FX3OImJIRKyIiJ9W7bqxWl70+YIu2zao5+ygGgOPiDOBHcC3MvOEatn/Al7KzAURMQc4IjM/N5h19lQn7ZoH7MjMLw1mbX0REaOB0Zn5k4g4HFgFXAxcScHnrIt2fYiCz1lEBHBYZu6IiEOAx4BZwAcp+HxBl207n0E8ZwdVDzwzHwVe2mvxRcDi6v1iav+QitJJu4qXmZsy8yfV+5eBtdTuAC76nHXRrqJlzY7q4yHVn6Tw8wVdtm1QHVQB3omjM3MT1P5hAaMGuZ5GmhkRT1dDLMX92NpeRDQDJwE/5gA6Z3u1Cwo/ZxHRFBGrgS3AQ5l5wJyvTtoGg3jODPAD198B/xVoATYBNw1qNX0QEcOAu4FPZ+bvBrueRumgXcWfs8x8PTNbqN2ZPSkiThjkkhqmk7YN6jkzwGFzNSbZNja5ZZDraYjM3Fz9B/cH4BvUZo8sTjXeeDdwR2b+c7W4+HPWUbsOlHMGkJm/BR6mNkZc/Plqr33bBvucGeC12/+nV++nA/cMYi0N0/YPpvLnwJrO1t1fVb84+gdgbWZ+ud1XRZ+zztpV+jmLiJERMbx6PxQ4F1hH4ecLOm/bYJ+zg+0qlCXAFGrTQG4GPg98G7gT+DPgV8C0zCzqF4KdtGsKtR/rEmgFrmkbhyxFRLwH+BHwDPCHavH11MaLiz1nXbTrMgo+ZxFxIrVfUjZR6xzemZnzI+IoCj5f0GXbbmMQz9lBFeCSdCBxCEWSCmWAS1KhDHBJKpQBLkmFMsAlqVAGuPYLEfG3EfHpdp8fiIhvtvt8U0T89z4eY0pETO7kuysjYms1o9y6iJjdw303t80GGRETIuIrfalVqocBrv3FvwKTASLiDdSuaT++3feTgcfr2VFENHXy1ZS2Y3RiWXWr9LuBv46It3axbqcyc2Vmfqo320o9YYBrf/E4/z9cj6d2R9vLEXFERBwKHAc8FRHnRMRTEfFMNXnQoQAR0RoRcyPiMWBaRHwqIp6rJhlaWk0a9d+A2VUv+4zOCsnMbcAvgLbbv+dGxJMRsSYiFlV3UhIRp1TzQz8BXNu2fdXTv696Py8iPtPuuzVVb/2wiLi/2n5NRFzaoL9HHUR6/VBjqZEy84WI2BURf0YtyJ+gNsXq6cB24GlqHY5bgXMy8+cR8S3gL4GF1W52ZuZ7ACLiBWBsZv5nRAzPzN9GxN9Tx9zNVQ1DqmMCfDUz51ff3QZcAHwH+EfgrzLzkYj4Yg+bfD7wQmZOrfb7ph5uL9kD136lrRfeFuBPtPv8r8A7gA2Z+fNq/cXAme22X9bu/dPAHRFxBbCrzuNfGhHPAs8DN2fmzmr5WRHx44h4BjgbOL4K3OGZ+Ui1zm09aCfUbqM/NyK+EBFnZOb2Hm4vGeDar7SNg4+nNoTyb9R64G3j39HN9v/R7v1U4GvAKcCqiKjnp81lmXk8cAZwU0T8aUQMAb4OXJKZ46nNODekqqWeeSh2see/syEA1f+ETqEW5H8TEXPr2Je0BwNc+5PHqQ1PvFRN0fkSMJxaiD9BbWa75oh4W7X+XwCP7L2T6pegb83MHwKfrfYxDHgZOLy7IjLzCWo96llUgQu8WM3ffUm1zm+B7dXEVACXd7K7VuDkqq6TgbHV+zcDv8/M24Evta0j9YQBrv3JM9SuPvm3vZZtz8wXqyGNjwH/VA1n/AH4+w720wTcXq3zFPC3VeB+B/jz7n6JWflCdazXqfW6n6E2c+WT7db5GPC16peYr3Syn7uBI6P2JJe/BNqGf8YDK6rlfw38z27qkfbhbISSVCh74JJUKANckgplgEtSoQxwSSqUAS5JhTLAJalQBrgkFer/AebgZaL0HubPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "index = clf.feature_importances_.argmax()\n",
    "plt.hist(X[y == 0][:, index], label='Malignant', bins='auto')\n",
    "plt.hist(X[y == 1][:, index], label='Benign', color='y', bins='auto',\n",
    "         alpha=0.5)\n",
    "plt.xlabel('Worst Radius')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The histogram reveals an enormous separation between malignant and benign worst-radius measurements. By training a Decision tree, we gained insights into medicine and biology.\n",
    "\n",
    "## 22. 3 Decision Tree Classifier Limitations\n",
    "\n",
    "Over-memorization limits the usefulness of our trained models. This is referred to as **overfitting**. Decision tree classifiers are particularly prone to overfitting, since they are able to memorize the training data. For instance, our cancer detector `clf` has perfectly memorized the training set `(X, y)`. We can confirm by outputting the accuracy with which `clf.predict(X)` corresponds to `y`.\n",
    "\n",
    "**Listing 22. 55. Checking the accuracy of the cancer cell model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our classifier has memorized the training data with 100% accuracy.\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "accuracy = accuracy_score(clf.predict(X), y)\n",
    "print(\"Our classifier has memorized the training data with \"\n",
    "      f\"{100 * accuracy:.0f}% accuracy.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classifier can identify any training example with 100% accuracy. However, this does not mean it can generalize well to real-world data. We can better gauge the classifier's true accuracy using cross-validation.\n",
    "\n",
    "**Listing 22. 56. Checking model accuracy with cross-validation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The classifier performs with 90% accuracy on the validation set.\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, )\n",
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(X_train, y_train)\n",
    "accuracy = accuracy_score(clf.predict(X_test), y_test)\n",
    "print(f\"The classifier performs with {100 * accuracy:.0f}% accuracy on \"\n",
    "       \"the validation set.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classifier's true accuracy is at 90%. That accuracy is decent, but we can definitely do better.\n",
    "\n",
    "## 22. 4 Improving Performance using Random Forest Classification\n",
    "\n",
    "Sometimes, an the aggregated viewpoint of a crowd outperforms all individual predictions. This aggregated triumph of collective intelligence is called the wisdom of the crowd. Lets explore the wisdom of the crowd by initializing 100 Decision trees. A large collections of trees is referred to as a **forest**. \n",
    "\n",
    "**Listing 22. 57. Initializing a 100-tree forest**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "forest = [DecisionTreeClassifier() for _ in range(100)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How should we train the trees within the forest? Naively, we could train each individual tree on our cancer training set `(X_train, y_train)`. However, we would end up with 100 trees that memorized the exact same data. Instead, we need to randomize our training data, in order to make the trained trees more diverse. We can do this using Bootstrapping with Replacement, as described in Section Seven.\n",
    "\n",
    "**Listing 22. 58. Randomly sampling a new training set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "def bootstrap(X, y): \n",
    "    num_rows = X.shape[0]\n",
    "    indices = np.random.choice(range(num_rows), size=num_rows, \n",
    "                               replace=True)\n",
    "    X_new, y_new = X[indices], y[indices]\n",
    "    return X_new, y_new\n",
    "    \n",
    "X_train_new, y_train_new = bootstrap(X_train, y_train) \n",
    "assert X_train.shape == X_train_new.shape\n",
    "assert y_train.size == y_train_new.size\n",
    "assert not np.array_equal(X_train, X_train_new)\n",
    "assert not np.array_equal(y_train, y_train_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets run our `bootstrap` function 100 times in order to generate 100 different training sets.\n",
    "\n",
    "**Listing 22. 59. Randomly sampling 100 new training sets**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1)\n",
    "features_train, classes_train = [], []\n",
    "for _ in range(100):\n",
    "    X_train_new, y_train_new = bootstrap(X_train, y_train)\n",
    "    features_train.append(X_train_new)\n",
    "    classes_train.append(y_train_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Across our hundred training sets, the data may be different but all the features are the same. However, we can increase the overall diversity by randomizing the features in `features_train`.\n",
    "\n",
    "**Listing 22. 60. Randomly sampling the training features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Random features utilized by Tree 0:\n",
      "['concave points error' 'worst texture' 'radius error'\n",
      " 'fractal dimension error' 'smoothness error']\n",
      "\n",
      "Random features utilized by Tree 99:\n",
      "['mean smoothness' 'worst radius' 'fractal dimension error'\n",
      " 'worst concave points' 'mean concavity']\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "sample_size = int(X.shape[1] ** 0.5)\n",
    "assert sample_size == 5\n",
    "feature_indices = [np.random.choice(range(30), 5, replace=False)\n",
    "                   for _ in range(100)]\n",
    "\n",
    "for i, index_subset in enumerate(feature_indices):\n",
    "    features_train[i] = features_train[i][:, index_subset]\n",
    "    \n",
    "for index in [0, 99]:\n",
    "    index_subset = feature_indices[index]\n",
    "    names = feature_names[index_subset]\n",
    "    print(f\"\\nRandom features utilized by Tree {index}:\")\n",
    "    print(names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets train each _ith_ tree within `forest` on training set `(features_train[i], classes_train[i])`.\n",
    "\n",
    "**Listing 22. 61. Training the trees within the forest**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, clf_tree in enumerate(forest):\n",
    "    clf_tree.fit(features_train[i], classes_train[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've trained every tree within the forest. Now, lets put the trained trees to a vote. What is the class-label of the data-point at `X_test[0]`? We can check using the wisdom of the crowd.\n",
    "\n",
    "**Listing 22. 62. Using tree-voting to classify a data-point**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We counted 93 votes for class 0.\n",
      "We counted 7 votes for class 1.\n",
      "\n",
      "Class 0 has received the plurality of the votes.\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "feature_vector = X_test[0]\n",
    "votes = []\n",
    "for i, clf_tree in enumerate(forest):\n",
    "    index_subset = feature_indices[i]\n",
    "    vector_subset = feature_vector[index_subset]\n",
    "    prediction = clf_tree.predict([vector_subset])[0] \n",
    "    votes.append(prediction)\n",
    "\n",
    "class_to_votes = Counter(votes)\n",
    "for class_label, votes in class_to_votes.items():\n",
    "    print(f\"We counted {votes} votes for class {class_label}.\")\n",
    "    \n",
    "top_class = max(class_to_votes.items(), key=lambda x: x[1])[0]\n",
    "print(f\"\\nClass {top_class} has received the plurality of the votes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "93% of the trees have voted for Class 0. Lets check if this majority vote is correct.\n",
    "\n",
    "**Listing 22. 63. Checking the true class of the predicted label**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The true class of the data-point is 0.\n"
     ]
    }
   ],
   "source": [
    "true_label = y_test[0]\n",
    "print(f\"The true class of the data-point is {true_label}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The forest has identified the point at `X_test[0]`. Now, we will use voting to identify all points in the validation set. Afterwards, we'll measure our prediction accuracy.\n",
    "\n",
    "**Listing 22. 64. Measuring the accuracy of the forest model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The forest has predicted the validation outputs with 96% accuracy\n"
     ]
    }
   ],
   "source": [
    "predictions = []\n",
    "for i, clf_tree in enumerate(forest):\n",
    "    index_subset = feature_indices[i]\n",
    "    prediction = clf_tree.predict(X_test[:,index_subset])\n",
    "    predictions.append(prediction)\n",
    "\n",
    "predictions = np.array(predictions)\n",
    "y_pred = [Counter(predictions[:,i]).most_common()[0][0]\n",
    "          for i in range(y_test.size)]\n",
    "accuracy = accuracy_score(y_pred, y_test)\n",
    "print(\"The forest has predicted the validation outputs with \"\n",
    "      f\"{100 * accuracy:.0f}% accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By leveraging the wisdom of the crowd, we have managed to improve performance. In the process, we have also trained a **Random Forest classifier**. Random Forest classifiers are collections of trees whose training inputs are randomized to maximize diversity.\n",
    "\n",
    "## 22. 5 Training Random Forest Classifiers Using Scikit-Learn\n",
    "In Scikit-Learn, Random Forest classification is carried out by the `RandomForestClassifier` class. \n",
    "\n",
    "**Listing 22. 65. Training a Random Forest classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The forest has predicted the validation outputs with 97% accuracy\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "clf_forest = RandomForestClassifier()\n",
    "clf_forest.fit(X_train, y_train)\n",
    "y_pred = clf_forest.predict(X_test)\n",
    "accuracy = accuracy_score(y_pred, y_test)\n",
    "print(\"The forest has predicted the validation outputs with \"\n",
    "      f\"{100 * accuracy:.0f}% accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, Scikit-Learn's Random Forest classifier utilizes 100 decision trees. However, we can specify a lower or higher count using the `n_estimators` parameter. Below, we'll lower the number of trees to 10.\n",
    "\n",
    "**Listing 22. 66. Training a 10-tree Random Forest classifier**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 10-tree forest has predicted the validation outputs with 97% accuracy\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "clf_forest = RandomForestClassifier(n_estimators=10)\n",
    "clf_forest.fit(X_train, y_train)\n",
    "y_pred = clf_forest.predict(X_test)\n",
    "accuracy = accuracy_score(y_pred, y_test)\n",
    "print(\"The 10-tree forest has predicted the validation outputs with \"\n",
    "      f\"{100 * accuracy:.0f}% accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the 10 trees is assigned a random subset of five features. Every feature in the subset contains its own feature importance score. Scikit-Learn is able to average all these scores across all the trees. The aggregated averages can be accessed by calling `clf_forest.feature_importances_`. \n",
    "\n",
    "**Listing 22. 67. Ranking the Random Forest features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'worst perimeter' has an importance score of 0.20\n",
      "'worst radius' has an importance score of 0.16\n",
      "'worst area' has an importance score of 0.16\n"
     ]
    }
   ],
   "source": [
    "for i in np.argsort(clf_forest.feature_importances_)[::-1][:3]:\n",
    "    feature = feature_names[i]\n",
    "    importance = clf_forest.feature_importances_[i]\n",
    "    print(f\"'{feature}' has an importance score of {importance:0.2f}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
